articles / reproducibility

Same commit, same bytes: a reproducible builds checklist

A build is reproducible when the same source, built with the same declared toolchain and instructions, gives bit-for-bit identical output, no matter who builds it, when, or on which machine. The Reproducible Builds project has spent more than a decade pushing Linux distributions in this direction, and the techniques it popularised apply to any project that ships artifacts.

We care about it for three practical reasons. You can verify that a binary really came from the source it claims to. Content-addressed caches work much better when equal inputs give equal outputs, a point from the note on cache keys. And when two builds do differ, a reproducible baseline turns "something changed" into a diff you can actually read.

First, find out whether you have a problem

Build the same commit twice, in two different directories, and compare:

git clone --quiet . /tmp/rb-a
git clone --quiet . /tmp/rb-b
(cd /tmp/rb-a && ./build.sh)
(cd /tmp/rb-b && ./build.sh)
sha256sum /tmp/rb-a/out/app.tar.gz /tmp/rb-b/out/app.tar.gz

Different paths expose leaked build paths; different times expose leaked timestamps. If the hashes differ, diffoscope shows where: it recursively unpacks, decompresses and disassembles what it can, and prints a readable diff of only the parts that differ. reprotest automates the other half, building twice while deliberately varying the environment: time, time zone, locale, umask, build path, file ordering and more.

The usual suspects

Timestamps

The most common difference by far, and the reason SOURCE_DATE_EPOCH exists. The specification is short: if the variable is set, it holds a Unix timestamp, and tools that would otherwise embed the current time should use it instead. The usual value is the time of the last commit, because it changes when the source changes and at no other time:

export SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)"

Support is broad. GCC has honoured it for __DATE__ and __TIME__ since version 7, and both GCC and Clang can warn about those macros with -Wdate-time. Python's py_compile switches to hash-based .pyc invalidation when it is set, instead of recording the source file's mtime. Many documentation and packaging tools respect it as well. Where a tool doesn't, normalise its output afterwards.

Archives

tar and zip record file order, modification times, owners and permissions, and all four tend to differ between machines. With GNU tar the full incantation is:

tar --sort=name \
    --mtime="@${SOURCE_DATE_EPOCH}" \
    --owner=0 --group=0 --numeric-owner \
    --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
    -cf out/app.tar -C build .
gzip -9n out/app.tar

--sort=name removes the dependence on directory read order. --mtime gives every entry the same time; add --clamp-mtime if you only want to lower times that are newer than that. The --pax-option line keeps access and change times out of pax extended headers. And gzip -n stops gzip from writing the original file name and timestamp into its own header, a detail that catches almost everyone once.

zip has no sort option, so sort the file list yourself and set modification times first. Zip stores times as local time without a zone, which makes TZ part of the output:

find build -exec touch --no-dereference --date="@${SOURCE_DATE_EPOCH}" {} +
(cd build && find . -type f | LC_ALL=C sort | TZ=UTC zip -X -q -@ ../out/app.zip)

-X leaves out extra attributes such as Unix owner IDs and extended timestamps. When you don't control the tool that writes an archive, a JAR from a build plugin for example, Debian's strip-nondeterminism can normalise zip and JAR files after the fact.

File system order

Anything that lists a directory (a bundler discovering entry points, a code generator walking a tree, a script collecting object files) can see entries in whatever order the file system returns them. readdir, Python's os.listdir and find make no ordering promise at all. Shell globs are sorted, but by the current locale's collation, which is the next problem. Sort explicitly wherever order reaches the output.

Locale and time zone

Collation differs between locales: LC_ALL=C sorts by byte value and en_US.UTF-8 doesn't, so a "sorted" list can still differ between a laptop and a CI runner. Dates formatted for humans depend on TZ. Pin both at the top of the build script, together with the umask, which decides the permissions of every file the build creates and therefore what ends up in archives:

#!/bin/sh
set -eu
export LC_ALL=C TZ=UTC
export SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)"
umask 0022

If your tools need a UTF-8 locale, C.UTF-8 is available on most current Linux distributions and sorts by code point.

Build paths

Compilers embed source paths in debug information and in __FILE__. Remap them: -ffile-prefix-map=$PWD=. for GCC 8 or later and current Clang, -trimpath for Go, --remap-path-prefix for Rust.

Iteration order and other randomness

Generated code is a frequent culprit. Python randomises string hashing per process, so iterating over a set of strings can give a different order on every run, and Go deliberately randomises map iteration order. Anything a generator writes out from a set or a map should be sorted first. The same goes for random identifiers, temporary file names that leak into output, and parallel steps that append to a shared file in whatever order they finish.

Inputs that aren't pinned

Reproducibility is always relative to the inputs, so the inputs have to stay put: lockfiles for dependencies, a pinned toolchain version, and base images referenced by digest rather than by a tag that moves. A build that downloads "the latest" of anything is only reproducible by accident.

Containers

Recent BuildKit versions accept SOURCE_DATE_EPOCH as a build argument and use it for the timestamps in image metadata, and can optionally rewrite file timestamps inside layers as well. The image digest is only stable if the layer contents are deterministic, though, which brings you back to everything above.

Where to start

Don't fix everything at once. Pick the artifact you actually ship, make two builds of it agree, and add that comparison to CI so it stays fixed. Most differences diffoscope finds are one of the suspects above, often fixed in a single line. Then move on to the next artifact.

• SOURCE_DATE_EPOCH set from the last commit
• LC_ALL, TZ and the umask pinned
• Archives sorted, with fixed times and owners; gzip -n
• Build paths remapped in compiler output
• Generated files written in sorted order
• Toolchain, dependencies and base images pinned
• A two-build comparison running in CI