articles / ci

Where CI time actually goes

Most slow pipelines we look at aren't slow because of one big thing. They're slow because of a dozen medium things nobody measured, arranged along a critical path nobody drew. Before changing any YAML, it pays to know where the minutes go.

Measure phases, not pipelines

A single pipeline duration hides everything interesting. Break each run into phases (waiting for a runner, preparing the environment, building, testing, uploading results) and collect them across many runs. Look at the median and the slow tail separately: a reasonable median with a terrible 90th percentile usually means queueing or flakiness, not compute. Most CI systems expose per-step timings through their API; a few hundred runs in a spreadsheet will show the pattern.

Then draw the job graph and mark its critical path, the longest chain of jobs that wait on each other. Wall-clock time is the length of that chain plus queueing. Speeding up a job that isn't on it changes nothing anyone will notice.

Setup is a tax on every job

Every job starts on a clean machine and pays for checkout, toolchain installation and dependency restore before doing anything useful. That tax is why splitting a pipeline into many small jobs can make it slower rather than faster. Reduce it first.

• Shallow clones. Building and testing usually needs one commit, not the whole history, and a depth of one is already the default in some CI checkout steps. Jobs that compute changed files or run git describe need more; a blobless clone (--filter=blob:none) gives them full history while fetching file contents only when they are needed.
• Prebuilt images. If every job installs the same compilers and system packages, bake them into the job image and rebuild it on a schedule.
• Dependency caches keyed on lockfiles. Cache the package manager's download cache, not the installed tree, when the install step starts clean anyway. npm ci, for example, deletes an existing node_modules before installing, so caching node_modules next to it buys nothing. Cache ~/.npm instead.

Cache build outputs, not just downloads

Downloads are the easy part. The larger wins come from reusing compilation and test results across runs, which takes a content-addressed build cache: Bazel's or Gradle's remote cache, Nx or Turborepo with a shared cache, ccache or sccache for C, C++ and Rust. The keys need the care described in Cache keys are a correctness problem. The payoff is that a change to one package stops rebuilding all of them.

Container builds deserve a mention of their own, because ephemeral runners start with an empty layer cache. BuildKit can export its cache to a registry and import it on the next run:

docker buildx build \
  --cache-from type=registry,ref=registry.example.com/app:buildcache \
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max \
  --tag registry.example.com/app:"$GIT_SHA" \
  --push .

mode=max exports the layers of every stage, not only those in the final image. That matters for multi-stage builds, where the expensive work happens in a builder stage that never ships.

Parallelism, with the tax in mind

Independent work (linting, type checking, unit tests, building) can run as parallel jobs. It helps when each job's useful work is large compared with its setup; when it isn't, keep the steps in one job and use the machine's cores instead. Most test runners do that with a flag or by default: pytest -n auto with pytest-xdist, Gradle's maxParallelForks, Jest's worker pool, and go test, which runs packages in parallel out of the box.

Check that a job can use what you give it. One busy core on a large machine means it is waiting on something serial, often one slow test file or the dependency install, and a bigger machine won't help.

Split tests by time, not by count

When one test job is the critical path, sharding it across machines helps, provided the shards are balanced. Splitting by file count rarely is: a single integration suite can outlast a hundred unit test files. Split by recorded duration instead. Most runners can write JUnit XML with per-test timings; keep the last good report as a CI artifact and feed it to a splitter like this one:

import heapq

def split(timings: dict[str, float], shards: int) -> list[list[str]]:
    """Longest first: each file goes to the currently lightest shard."""
    heap = [(0.0, i) for i in range(shards)]
    buckets: list[list[str]] = [[] for _ in range(shards)]
    for name, secs in sorted(timings.items(), key=lambda kv: kv[1], reverse=True):
        load, i = heapq.heappop(heap)
        buckets[i].append(name)
        heapq.heappush(heap, (load + secs, i))
    return buckets

This is the classic longest-processing-time heuristic. Its result is provably never more than a third longer than the best possible split, which is more precision than real timing data deserves. Give files without a recorded timing the median duration; at zero they would all pile into the same shard.

If you'd rather not maintain a splitter, several runners shard natively. Jest and Playwright both accept --shard=1/4, though they balance by number of tests or files rather than by recorded time. In GitHub Actions, a matrix runs the shards side by side:

test:
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix:
      shard: [1, 2, 3, 4]
  steps:
    - uses: actions/checkout@v4
    - run: npm ci
    - run: npx jest --shard=${{ matrix.shard }}/4

Do less

The fastest job is the one that doesn't run.

• Cancel superseded runs. After a new push, the run for the previous commit is usually worthless; the GitHub Actions setting is below.
• Run only what a change affects. In a monorepo that means a task graph, which is a big enough topic for its own article.
• Fix or quarantine flaky tests instead of retrying them. Automatic retries slow down exactly the runs that are already struggling, and hide the signal you need.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Order of attack

1. Measure phases and find the critical path.
2. Cut setup: shallow clones, prebuilt images, dependency caches.
3. Add a build cache for compilation and tests.
4. Parallelise where the work outweighs the setup, and split tests by time.
5. Stop running what doesn't need to run.

Re-measure after each step. The critical path moves, and whatever was second-worst becomes the thing that matters.