Why “it feels faster” is not good enough
Pipeline speed work is full of anecdotes. Someone adds a cache and says builds are quicker. A week later a large dependency update lands and they feel slow again. Without measurement, you cannot tell whether the cache helped, whether a runner change hurt, or whether the numbers moved because of unrelated factors such as load and test growth.
A benchmark turns opinion into evidence. It also helps a CTO decide whether further investment is worth it. This page describes how we run before and after comparisons as part of our CI/CD pipeline engagements, and what honest reporting looks like. We do not publish generic claims about how much faster pipelines get, because results depend entirely on your starting point.
What good measurement looks like
- A baseline taken first. Data from before any change, covering enough runs to include slow days and big changes.
- The right statistics. Median for the typical case, 95th percentile for the painful case.
- Components separated. Queue time, setup time, build, test and deploy are measured individually.
- One change at a time. Each optimisation has its own measurement.
- Comparable conditions. Cold and warm cache runs are never mixed in one comparison.
- Cost included. Faster is only better if you know what it cost.
What we measure
| Metric | Why it matters |
|---|---|
| Median pull request check duration | The typical wait a developer experiences |
| p95 duration | The slow runs people remember and complain about |
| Queue time | Waiting for a runner, a capacity problem rather than a code problem |
| Time per stage and step | Locates the critical path |
| Failure rate excluding genuine test failures | Infrastructure flakiness and unreliable tests |
| Retry rate | Hidden cost of flakiness |
| Cache hit rate | Whether caches actually work |
| Runner minutes and cost | The price of speed |
| Merge to production time | End-to-end lead time, an important DORA measure |
The last item connects pipeline speed to delivery performance. Lead time, deployment frequency, change failure rate and time to restore are the four DORA metrics. They are useful as trends. They are not a scoreboard to rank engineers.
How we collect data
Most CI platforms expose what we need. GitHub provides workflow run and job data through its API, GitLab provides pipeline and job timings, and Jenkins exposes build data and metrics through plugins. A short script can pull this into a spreadsheet or database. For example, listing recent workflow runs with GitHub’s CLI and computing durations:
gh run list --workflow ci.yml --limit 200 \
--json databaseId,createdAt,updatedAt,conclusion,event \
| jq -r '.[] | select(.event=="pull_request")
| [.databaseId, .conclusion,
((.updatedAt|fromdateiso8601) - (.createdAt|fromdateiso8601))]
| @csv'
createdAt to updatedAt approximates total run time including queueing, so for finer detail we pull per-job startedAt and completedAt values. The output feeds a median and p95 calculation. For ongoing monitoring, we send pipeline events to Prometheus and chart them in Grafana, as described in our dashboard work, so the benchmark becomes a permanent view rather than a one-off report.
Finding the critical path
A pipeline is a graph of jobs. Total duration equals the longest path through it. Speeding up a job that runs in parallel with a longer one achieves nothing.
We draw the graph with measured durations, then attack the longest path first. Typical findings:
- A single slow end-to-end test job dominating everything else, which suggests sharding or moving it off the pull request path.
- Long setup phases dominated by dependency downloads and image builds, which suggests caching.
- Queue time exceeding run time, which points at runner capacity rather than the pipeline.
- Serial jobs that do not depend on each other and could run side by side.
- Heavy jobs running on every change when path filters could skip them.
Running the experiment properly
For each optimisation:
- State the hypothesis (“caching npm will save about two minutes of install time”) before the change.
- Ship the change on its own, ideally on a branch run against recent commits for a controlled comparison.
- Compare equivalent runs: similar change sizes, same runner type, and the same cache state.
- Look at the median and p95, and at failure rate, since a faster but flakier pipeline is not an improvement.
- Keep, adjust or revert the change based on the result.
A subtle trap is regression to the mean: a measured improvement over a few lucky runs. We use enough runs and look at the distribution, not individual numbers. Another is the “cold start” illusion, where the first run after a cache change is slow and the next is fast. We record both.
Decisions and trade-offs
Speed versus cost. Larger runners and more shards cut wall-clock time and may raise spend. A fair report shows both so you can decide whether five minutes of developer waiting is worth the extra runner cost. For a large team it often is.
Speed versus coverage. The quickest way to a fast pipeline is to test less. We do not recommend that for production paths. We prefer moving slow, low-value tests to scheduled runs and keeping critical checks on every pull request.
Hosted versus self-hosted runners. Self-hosted runners may be faster per job on better hardware, but add maintenance. We benchmark both for your heaviest jobs before recommending a change.
When the benchmark says do nothing. If your median check time is already modest, queue time is low and failures are rare, the right advice is to leave it alone and put effort elsewhere. We would rather say that than sell work.
Common failure modes
- No baseline, so improvement cannot be shown.
- Averages that hide slow runs.
- Measuring warm runs only.
- Changing many things at once, so nobody knows what worked.
- Ignoring queue time, and blaming the pipeline for a capacity shortage.
- Optimising a non-critical job.
- Reporting cherry-picked best runs.
Delivery and handover
You receive a report with the baseline, each change we made and its measured effect, the methodology (so your engineers can reproduce it), and a ranked list of remaining opportunities. The collection scripts and dashboards are left in your repositories. If your migration from another CI system is part of the work, the same method gives a fair comparison, which we apply in our Jenkins migration approach.
How to tell it is working
The sign that this approach works is that numbers keep being collected after we leave. Median and p95 check durations stay inside agreed bounds, regressions are noticed within days rather than quarters, and slowdowns trigger a conversation. If you would like a baseline for your pipeline, book a consultation and we will start with the data.