Muhammed Senussi
Muhammed Senussi
  • Muhammed Senussi
Your CI Is Too Slow and It Is Changing How People WorkDevOps

Pipeline duration is usually tracked as an infrastructure metric. It is really a behavioural one.

What slow CI actually buys you

  • Bigger commits, because nobody wants to pay the wait twice.
  • Longer-lived branches, and therefore harder merges.
  • Approvals given without a second look, because a re-run is a coffee break.
  • A quiet norm of pushing to main on Friday afternoon, since the pipeline will not finish before anyone leaves anyway.

Where the time usually goes

In the pipelines I have profiled it is rarely the tests. It is dependency resolution with a cold cache, container image builds that reinstall the world because a layer was ordered badly, and stages running in sequence that have no dependency on one another. Fixing the layer order alone has repeatedly cut builds in half.

Set a budget, put the duration on a dashboard, and treat a regression the way you would treat a latency regression in production. It is the same kind of problem.

Your CI Is Too Slow and It Is Changing How People Work — Figure 1
Your CI Is Too Slow and It Is Changing How People Work — Figure 2
Your CI Is Too Slow and It Is Changing How People Work — Figure 3

2 Comments

  • Peter Bakker

    February 13, 2024

    The Friday afternoon observation is painfully familiar. We measured and our median PR wait was 26 minutes.

    • Muhammed Senussi

      Muhammed Senussi

      AuthorFebruary 13, 2024

      Twenty-six is right in the danger zone. Under ten and people stay in the loop; over twenty they go and do something else and the review culture changes without anyone deciding to change it.

Leave a comment