articles / monorepos

Task graphs: rebuilding only what changed

A monorepo stays pleasant to work in only if the build doesn't grow with it. The tools that make that possible (Bazel, Buck2, Pants, Gradle, Nx, Turborepo and others) differ in many ways, but underneath they share one structure: a task graph, and rules for deciding which parts of it have to run.

From packages to tasks

Start with the package graph. Nodes are packages and edges mean "depends on". Each package has tasks such as build, test and lint, and tasks have dependencies of their own: building app requires building ui first, testing ui requires building ui, and linting needs nothing. The task graph combines the two, and tasks with no path between them can run in parallel.

In Turborepo the structure is written down in turbo.json, where ^ means "this task, in my dependencies":

{
  "globalDependencies": [".nvmrc", "tsconfig.base.json"],
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["$TURBO_DEFAULT$", "!README.md"],
      "outputs": ["dist/**"],
      "env": ["NODE_ENV"]
    },
    "test": {
      "dependsOn": ["build"]
    },
    "lint": {}
  }
}

Nx expresses the same thing with dependsOn and named inputs, Gradle with task dependencies between projects, and Bazel with targets and their declared deps. The vocabulary differs; the graph is the same.

Two ways to skip work

Caching asks, for each task, whether this exact set of inputs has been seen before. The task's key covers its own source files, its configuration, the relevant environment and the keys of the tasks it depends on: the construction from the article on cache keys, applied node by node. A change to tokens changes the key of its build, which changes the key of the ui build, and so on up the graph. Nobody has to reason about that propagation; it falls out of the hashing.

Affected detection asks, before anything runs, which packages a change could possibly influence, and leaves the rest of the graph out entirely. The usual implementation lists the files changed since the merge base, maps each file to the package that owns it, then walks the graph backwards to collect every package that depends on those, directly or transitively.

git diff --name-only origin/main...HEAD

The three-dot form compares against the merge base, so commits that landed on main after you branched don't count as yours. Walking the graph backwards is a breadth-first search over reversed edges:

from collections import deque

deps = {
    "app": ["ui", "api-client"],
    "admin": ["ui"],
    "ui": ["tokens"],
    "api-client": [],
    "tokens": [],
}

def affected(changed: set[str]) -> set[str]:
    dependents = {pkg: set() for pkg in deps}
    for pkg, ds in deps.items():
        for d in ds:
            dependents[d].add(pkg)
    seen, queue = set(changed), deque(changed)
    while queue:
        for parent in dependents[queue.popleft()]:
            if parent not in seen:
                seen.add(parent)
                queue.append(parent)
    return seen

print(sorted(affected({"tokens"})))  # ['admin', 'app', 'tokens', 'ui']

In CI this needs enough history to find the merge base. A clone with --depth=1 doesn't have it, which is the most common reason affected detection either fails outright or quietly falls back to building everything.

Why affected detection misses things

Affected detection is only as good as the graph. Its failure mode is a false "not affected", the graph-level twin of an undeclared cache input, and the causes are familiar.

• Files that belong to no package. Lockfiles, the root TypeScript config, toolchain version files, CI configuration and shared scripts influence everything but live outside every package. Treat a changed file with no owner as affecting all packages. It is conservative, and conservative is the right default.
• Undeclared dependencies. With a hoisted node_modules, a package can import a library, or a sibling package, that it never declared, and it works by accident. The graph doesn't know about that edge. pnpm's default layout makes most such imports fail, and lint rules like import/no-extraneous-dependencies catch many of the rest at review time.
• Non-code inputs. Tests that read fixtures from another package, generators that read a schema from a shared directory, anything that loads files through a computed path. Declare these as inputs, or move the files into the package that uses them.
• Undeclared outputs. A cache hit restores a task's declared outputs and nothing else. A task that also writes somewhere else works on a miss and is subtly broken on a hit.

Granularity and early cutoff

Package-level graphs are coarse: change one file in a large package and every task in it runs again, along with everything downstream. File- or target-level graphs, as in Bazel, Buck2 and Pants, can be far more precise, at the price of declaring much more of the build, by hand or with generators. For many JavaScript and TypeScript monorepos, package level is the right place to start, and splitting an oversized package is usually cheaper than adopting a finer-grained build system.

The other difference is what a task's key is built from. Bazel keys each action on the contents of its input files, so when an upstream action re-runs but produces byte-identical output, everything downstream still hits the cache. That property is called early cutoff. For Java it goes further: dependents compile against interface jars that contain only signatures, so changing a method body doesn't recompile them. Tools that hash a dependency's sources rather than its outputs don't get early cutoff, and every tool loses it when outputs aren't reproducible.

Sharing the cache safely

A shared remote cache multiplies the benefit: CI fills it and laptops reuse CI's results. Decide who is allowed to write. The usual arrangement is that only CI writes, from trusted branches, while developers only read. Bazel's --remote_upload_local_results=false exists for exactly that, and many hosted caches offer read-only tokens. A cache anyone can write to is a cache anyone can poison, on purpose or just with a dirty working tree.

Keeping the graph honest

Run the whole graph from scratch on a schedule (nightly is common) with the cache and affected filtering both switched off, and compare test results, and outputs where you can, with what the incremental pipeline produced. A failure there that the incremental runs didn't see points straight at a missing edge or an undeclared input, which is much cheaper to find on a quiet night than in the middle of a release.