The wait that costs the most
Developers can tolerate a slow deploy. They cannot tolerate a slow pull request check, because it sits in the middle of their day. A forty minute test run produces context switches, abandoned branches and large, risky pull requests. The usual response is “we need faster tests”, which is true but slow to achieve. Running the existing tests in parallel is often the quickest way to cut feedback time while the suite is improved over months.
This page covers how we set up parallel test execution and matrix builds as part of our CI/CD pipeline work, and where the approach stops helping.
Two different ideas that share a name
Sharding splits one test suite across several runners. It reduces wall-clock time. The same tests run exactly once, just concurrently.
Matrix builds run the same job across a set of variables: language versions, operating systems, database versions, browsers. They increase coverage and, by running in parallel, do not add wall-clock time. They multiply compute.
Teams mix these up, then wonder why a three-by-three matrix made CI slower and more expensive. We design them separately.
What good looks like
- Balanced shards. The slowest shard finishes close to the fastest.
- Independent tests. Any test can run in any order, on any runner, alongside any other.
- Readable results. Failures from all shards appear in one report with the failing test name, not buried in twelve logs.
- Bounded cost. Concurrency is capped and the shard count is chosen deliberately.
- Fast gates first. Linting and unit tests fail the build before any expensive parallel stage starts.
How sharding works in practice
Balance by time, not by count
If you divide 400 test files evenly across four runners, you rarely get four equal shards, because test files vary in duration by orders of magnitude. The slowest shard decides the finish time. A timing-based splitter reads stored durations from previous runs and distributes files so totals are similar. Tools such as pytest-xdist and plugins like pytest-split, Jest’s --shard option, Playwright’s --shard flag and commercial splitters all support variations of this. Timing data is stored as a cache or artefact between runs.
A GitHub Actions example
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
cache: npm
- run: npm ci
- run: npx playwright test --shard=${{ matrix.shard }}/4
- uses: actions/upload-artifact@v4
if: always()
with:
name: report-${{ matrix.shard }}
path: playwright-report/
Setting fail-fast: false is deliberate here: when one shard fails, you usually want to see every failure rather than stop at the first. In GitLab CI the equivalent is parallel: with the CI_NODE_INDEX and CI_NODE_TOTAL variables, and in Jenkins a parallel block of stages. For GitHub Actions, a later job with needs: test can merge reports.
Fix shared state before adding runners
The biggest surprise in parallelisation is the tests that start failing. Typical causes:
- A single test database used by every runner, with tests that assume a clean table.
- Fixed ports in integration tests.
- Tests writing to a shared temporary directory.
- Order-dependent tests that rely on earlier tests creating data.
The fix is isolation. A database per shard, started as a service container or through Testcontainers, removes most conflicts. Randomised port allocation solves the rest. We treat this as part of the work, because adding shards on top of non-isolated tests just makes flakiness faster.
Matrix builds: use them for compatibility, not speed
A matrix is right when you genuinely support several runtimes. A library supporting Python 3.10 to 3.13 should test all four. An internal service deployed on one runtime should test one.
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
node: [20, 22]
exclude:
- os: windows-latest
node: 20
The exclude list matters. Every combination is a job that costs minutes and can flake. We keep the matrix small on pull requests and run the wider combinations on merge to main or on a schedule.
Merging results and gating on them
Sharding creates a reporting problem: five jobs now produce five partial reports. Each shard should upload its JUnit XML (or equivalent) as an artefact, and a final job downloads them all and publishes one combined summary. That job is also the right place to gate merges. Make the required status check the single aggregate job, not each shard, so changing the shard count never means editing branch protection rules.
report:
needs: test
if: always()
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
pattern: junit-*
merge-multiple: true
The if: always() line is what makes the aggregate job run even when a shard fails, so the person reviewing the pull request sees every failure at once instead of fixing them one run at a time.
Decisions and trade-offs
Per-shard overhead. Each shard repeats checkout, dependency restore and sometimes container start. If that takes two minutes and your tests take three, four shards will not help much. Pair this with build caching to shrink the setup.
Runner capacity. Fifty simultaneous jobs from one pull request can starve other developers. Concurrency limits and queue monitoring protect everyone.
Flaky tests. Parallelism does not cause flakiness, but it reveals it. Retrying failed tests hides the problem. We quarantine known flaky tests, report them separately and fix root causes.
When not to parallelise. If the suite takes under five minutes, simplify first. If the slowness comes from one huge end-to-end test, splitting that test or moving it to a nightly job is better than adding runners.
Common failure modes
- Unbalanced shards from file-count splitting.
- Hidden shared state causing intermittent failures.
- Uploading nothing on failure. Without
if: always()the failing shard’s report is lost. - Matrix explosion from every version and OS combination.
- Unlimited concurrency, with queues so long that parallelism makes the wall-clock time worse.
Delivery and handover
We start by measuring the suite, fix isolation issues in cooperation with your developers, then introduce sharding and a merged report. The shard count and matrix definitions live in a shared workflow, so a new service inherits sensible defaults, as covered in reusable pipeline templates. You receive notes on how timing data is stored and how to adjust shard counts.
How to tell it is working
Watch median and 95th percentile pull request check duration, shard duration spread (slowest versus fastest), flake rate, and queue time. The target is a short, predictable check. We report the results as part of our before and after benchmarks.
If your test suite is the bottleneck, get in touch and we will look at the numbers with you.