Builds and CI

These are Inky Labs’ best practices for building a project: every repository follows them, and inkylabs-audit checks the ones that can be checked. Where a project departs from one, its AGENTS.md says so and why.

Partly set up for Inky Labs: the shared cache, the CI image, the AWS roles and the inkylabs/forks group exist (since 2026-10-26), but no pipeline has run with them yet, the group has one runner of its own, and no Inky Labs repository other than this handbook and plans.inkylabs.com builds with Bazel. The rules, and the measurements that justify them, come from SVRBC’s handbook, whose doreancon.org piloted them; where this page cites doreancon.org, it is SVRBC’s.

The rules

Every repository builds with Bazel, at a version pinned in .bazelversion. bazel build //... and bazel test //... work the same way in every repo.
Tools are pinned by the repository, never assumed from the machine, and fetched only when a step runs. Programs a build runs come from one pinned nixpkgs commit through inkylabs_bazel’s nix_tools (see below), not rules_nixpkgs. Upgrading a tool is then a one-line commit, and Bazel rebuilds exactly what depends on it.
CI uses one shared base image. It contains Nix, Bazelisk, the AWS CLI and bazel-cache-sync, and nothing project-specific. No repository maintains its own CI image. It is itself a Bazel step (below), built from the same Nix pins.
Everyone reads the shared cache; only Inky Labs’ CI and trusted workstations write to it. CI means pipelines in inkylabs/forks and tags (and staging branches) of the projects; a trusted workstation is an administrator’s (see below), and inkylabs-mr builds, tests and uploads from it before it pushes. A change is built once, and nobody builds it again: not its pipeline, not master (which runs no pipeline), not a reviewer, not another developer.
Secrets live in GitLab CI/CD variables, never in a repository. The cache itself needs none.
The same commands build every repository, here and in CI (see below). An npm script or a Makefile target may be a convenience alias, but docs and CI use the Bazel commands.
Cache every step that can be cached, including the expensive ones at the end: building the site itself, and anything rendered from it (such as PDFs printed from its pages). A step outside Bazel runs on every build, on every machine, forever.
Tests are Bazel tests. Every test a repository has runs under bazel test //..., so its result is cached like any step’s: a commit tested once, anywhere, is never tested again. A test runner called directly (npm test, cargo test) is a convenience for iterating, never what CI runs. Wrap the suite in an sh_test that lays out what it reads, as SVRBC’s doreancon.org’s //:unit_tests does for vitest.

Tools are fetched only when a step runs

A build whose every step comes from the cache should download no tools at all. The usual way to bring Nix packages into Bazel, rules_nixpkgs, can’t do that: it makes each tool an external repository and runs nix-build when the repository is fetched, and Bazel fetches every repository a target refers to before it knows which steps are cache hits. SVRBC’s doreancon.org’s first release through Bazel spent about 30 of its 279 seconds downloading MuseScore, Qt and the rest for steps that never ran.

So Inky Labs repositories use the nix_tools extension from this handbook’s bazel/ module instead:

One nix-instantiate evaluation writes every tool’s recipe (a .drv file) to the Nix store and records its path. Nothing is downloaded but nixpkgs itself.
A step that uses a tool fetches it as it runs: nix_tool("musescore") expands to $(realise.sh <drv>), the tool’s store path. A cache hit never runs, so it never fetches.
The .drv path is a hash of the tool’s exact recipe: the same on every machine, and written into the step’s command, so it is part of the cache key. A new pin is a new key.

bazel_dep(name = "inkylabs_bazel", version = "0.0.0")
git_override(
    module_name = "inkylabs_bazel",
    remote = "https://gitlab.com/inkylabs/developers.inkylabs.com.git",
    commit = "<a commit of developers.inkylabs.com>",
    strip_prefix = "bazel",
)

nix_tools = use_extension("@inkylabs_bazel//nix:extensions.bzl", "nix_tools")
nix_tools.nixpkgs(commit = "<nixpkgs commit>", sha256 = "<nix-prefetch-url --unpack hash>")
nix_tools.tool(name = "musescore", attr = "musescore")
nix_tools.tool(name = "fonts", file = "//nix:fonts.nix")   # your own expression
use_repo(nix_tools, "nix_tools")

load("@nix_tools//:tools.bzl", "NIX_TOOLS", "nix_tool")

genrule(
    name = "engraving",
    srcs = ["hymn.musicxml"],
    outs = ["hymn.svg"],
    tools = NIX_TOOLS,
    cmd = nix_tool("musescore") + "/bin/mscore -o $@ $<",
)

Three rules keep it honest:

Never name a tool by a host path or a bare command name. A step that needs a program on PATH (a script calling ffprobe) sets PATH=<pinned tool>/bin and nothing else, so a missing tool fails instead of quietly using the machine’s. SVRBC’s doreancon.org found one such leak: Ubuntu’s own ffprobe was standing in until the step was tested in a clean container.
Fetch tools before changing HOME. realise.sh finds Nix in the ordinary places; a step that swaps in a throwaway HOME does so after.
Test a cached build in a fresh container. The proof that nothing is fetched needlessly is an empty Nix store that stays empty: in SVRBC’s doreancon.org test, a fully cached build (94 steps) took 26 s and left no tools in the store, while editing one hymn then fetched MuseScore, and only it.

In CI, steps have to be able to write to /nix to fetch: the base image sets that up (/etc/bazel.bazelrc, with Nix on the plain PATH Bazel gives a step).

The standard commands

bazel build //...      # build every cacheable output
bazel test //...       # run every test Bazel knows about
bazel run //:assets    # put generated files where the app reads them, and run
                       # the steps that can't be cached

bazel build //... is the build. Every step that turns committed inputs into outputs is a Bazel action, so it’s cached and shared: engraving a hymn, probing a track’s length, rendering a PDF page to an image, and (the goal, rule 7) building the site and rendering from it.

bazel run //:assets exists because most apps read generated files from the source tree (Nuxt serves public/), while Bazel keeps outputs under bazel-bin/. In a repository with generated files, //:assets is a sh_binary that:

copies what Bazel built into the tree, overwriting in place (CI’s checkout belongs to the runner, so files there can be written but not chmod-ed);
then runs the steps that can’t be Bazel actions, such as rendering pages from a running dev server or anything that needs the npm dependencies, with tools Bazel pins (passed as $(rootpath …) arguments) rather than ones from the machine’s PATH.

Keep the second part small, and shrinking: anything that reads only committed files and writes files belongs in the first, and the second exists only until a step has been brought into Bazel. SVRBC’s doreancon.org’s BUILD.bazel and scripts/bazel/assets.sh are the worked example.

Tools outside Bazel come from a shell.nix at the repository root, pinned to the same nixpkgs commit as MODULE.bazel: typically Node for npm ci and a dev server. The tests are not among them (rule 8); a job that needs nix-shell is running something outside Bazel. Use mkShellNoCC unless something really compiles, so CI doesn’t download a C compiler. Run commands as nix-shell --run '…', or open a shell with nix-shell.

Each repository’s AGENTS.md names any extra command a person needs (such as npm ci first) and nothing else.

How the cache decides what to rebuild

Bazel models a build as actions. An action is one command with declared inputs and outputs, such as “engrave psalm-82 with MuseScore into an SVG.” The cache has two halves:

cas/ (content-addressable store): files, each stored under the SHA-256 of its own bytes. A blob can’t be stored under the wrong name, and identical files are stored once.
ac/ (action cache): results, each stored under the hash of the action itself: its command line, its environment, the hash of every input file and of every tool. A result says which outputs the action produced, by their cas/ hashes.

If an action’s ac/ entry exists, Bazel downloads its outputs instead of running it. Timestamps play no part, so a fresh clone on another machine gets the same hits.

Bazel also stops early. If an action re-runs but produces byte-identical output, the hashes downstream don’t change, so nothing downstream re-runs.

Caching is only correct if every input is declared. Bazel runs actions in a sandbox that hides undeclared files, so a missing input fails the build instead of producing a wrong cache hit.

Why only trusted machines write: a cas/ blob checks itself, but an ac/ entry is a claim (“this action produced that file”) that nobody can check without re-running the action. A bad writer could point a legitimate action at a poisoned file, and every reader would use it.

The shared cache

One cache serves every Inky Labs project. Keys hash the exact command, tools and inputs, so projects can’t collide, and identical work is stored once. There is no cache server:

Reading. Every repository’s .bazelrc points Bazel at the cache over plain HTTPS, and tells it never to upload:
Terminal window
```
common --remote_cache=https://bazel-cache.inkylabs.com
common --remote_upload_local_results=false
common --disk_cache=~/.cache/bazel-disk
```
CloudFront serves the private bucket read-only. A CloudFront function passes only ac/<sha256> and cas/<sha256> paths, so the bucket can’t be listed, and compression is off because Bazel checks every blob against its hash. Reads are public. That’s fine for public repositories, whose outputs anyone could build from the source; finding a private repository’s outputs would take its source too.
Writing. Bazel can’t write to S3 (it can’t sign requests), and doesn’t need to. A trusted machine builds into its disk cache as usual, then runs bazel-cache-sync, which uploads the new entries: every cas/ blob before any ac/ entry, so a result never appears before its outputs.
Retention. S3 lifecycle rules expire ac/ entries after 60 days and cas/ blobs after 67, so a result rarely outlives its outputs. When one does, Bazel notices the missing blob, re-runs that one action and carries on (SVRBC tested this with Bazel 9.2, in both download modes).
Misses are quiet. CloudFront may read the bucket’s key list, so a missing key is a 404 (a miss), not a 403 (an error). Listing itself is still blocked by the path filter.

Its OpenTofu configuration is in infra/bazel-cache/. Not yet applied for Inky Labs: the bucket, the distribution, the bazel-cache.inkylabs.com record and the writer role don’t exist yet.

Trusted workstations

A workstation is trusted when it belongs to an Inky Labs administrator, has full-disk encryption, and signs in to AWS through the access portal. There’s no shared secret to hand out: write access is the person’s own AWS role, and revoking it is removing their access.

It uploads when it opens a merge request: inkylabs-mr builds and tests the commit (bazel test //...) and runs bazel-cache-sync before it pushes, so the merge request’s pipeline finds every step done (see Getting started for the one-time setup). By hand, it is:

aws sso login --profile inkylabs
AWS_PROFILE=inkylabs bazel-cache-sync

Upload only what you built from committed code, as inkylabs-mr does: it refuses a working tree with changes in it, and uploads only what that build ran. It builds in an output base of its own with an empty disk cache, so the disk cache it uploads holds this commit’s new results and nothing else. Your everyday disk cache holds whatever you have built from uncommitted work too (4.4 GB of it, on the first SVRBC workstation measured, against 156 MB for the commit), and bazel-cache-sync by hand uploads all of it. An upload is trusted by every other machine, but no more than a fork’s pipeline is, which uploads a branch before anyone has reviewed it; review decides what merges, not what is cached. An entry says only that these inputs gave these outputs, and a commit nobody merges is simply never asked for again.

CI

Every repository includes the shared template rather than writing its own cache and runner setup:

include:
  - project: inkylabs/developers.inkylabs.com
    file: ci/bazel.yml

check:
  extends: .inkylabs-bazel
  script:
    # Builds every target under //..., not only the tests, and runs the tests.
    - bazel test //... --remote_download_minimal --test_output=errors
  rules:
    - if: $CI_COMMIT_BRANCH

It runs on every branch: in the fork, where a merge request is built, and on master after the merge. That is the whole job, even in a project with generated assets and npm tests (SVRBC’s doreancon.org): everything it checks is a Bazel step (rule 8), so once inkylabs-mr has uploaded a commit, the job finds it all cached and only confirms it.

A release job builds nothing: it fetches the deployable site, which check already built and cached, and deploys it (Releases, below).

.inkylabs-bazel runs the job in the shared base image. Its after_script asks GitLab for an OIDC token, exchanges it for short-lived AWS credentials, and runs bazel-cache-sync. The inkylabs-bazel-cache-writer role decides who may: any branch of a project in inkylabs/forks, and master, main and tags of inkylabs projects. A fork outside Inky Labs carries its own project path in that token, so it can’t. A failed upload is reported but never fails the job: the cache only ever saves time.

Do developers need to build locally? Their machines read the shared cache, so they build just the steps their change touches, and inkylabs-mr does that before it pushes. For something too heavy to build locally, inkylabs-mr --no-build pushes without it, and the fork’s pipeline builds it once for everyone. Building twice (locally and in CI) is never needed for consistency: a cached step is as good as a rebuild. It is about trust: what a workstation builds stays on it unless a trusted one uploads it.

Will CI reuse what I built locally? Only once it is in the shared cache. A build lands in the workstation’s own disk cache, which no pipeline can see, so until a trusted workstation uploads it (which inkylabs-mr does), the fork’s pipeline builds it all again. Uploaded, it doesn’t: cache keys come out the same on every machine (at SVRBC, a fresh doreancon.org build on a workstation found 94 of its 96 steps already cached by CI), so every step CI would run on the same inputs is a hit. Two things can still cost time. A pipeline still waits for a runner; and whatever a job does outside Bazel runs every time, which is why the tests are Bazel tests (rule 8). An untrusted machine’s merge requests are built by the fork’s pipeline, as before.

A step that isn’t reproducible makes this worse. Keys are computed from inputs, so two machines that each build a step get the same key but may get different bytes, and then everything downstream differs too. SVRBC met one in MuseScore, which writes a hymn SVG’s lyric extender lines in no fixed order, so a hymn engraved on a workstation differs from CI’s, invisibly, and the bulletin PDFs printed from it get new hashes. Make a step’s output stable (sort what the tool leaves unsorted) rather than relying on whoever built it first.

Runners

Every job runs on the group’s own runners, GitLab Runners Inky Labs keeps running, registered to the inkylabs group and taking untagged jobs. A job that only GitLab’s hosted runners can run is a bug: their free compute minutes run out, and then the job can’t run at all. SVRBC learned this on 2026-10-08, when its base image’s build, then pinned to them because rootless BuildKit needs privileges a group runner doesn’t give, stopped with ci_quota_exceeded mid-month. Both images, the CI one and the review bot’s, are Bazel steps now (Nix and nix2container), built on a group runner or a workstation and only uploaded by CI. Hosted runners may still pick up a job when they have minutes, but nothing may depend on them.

Inky Labs keeps at least two of its own runners online, on different machines, each taking several jobs at once (its concurrent setting above 1). With one runner taking one job at a time, every pipeline waits behind every other (eight deep at SVRBC on 2026-10-08), and CI stops whenever that one machine is down.

Not yet met for Inky Labs: the inkylabs group has one runner of its own, frankenstein (online on 2026-10-26), so a second is still needed.

inkylabs-audit warns when fewer than two of the group’s runners are online (a runner’s concurrency is in its own configuration, which GitLab’s API doesn’t show), and about a CI file that pins a job to GitLab’s hosted runners (saas-linux-… tags).

Verified commits

A fully cached bazel test //... still takes most of a minute, because Bazel can only find a step’s cache key by loading the whole build: starting, evaluating nixpkgs for the tool pins, analysing every target. None of that is needed to learn that a commit has already passed. So once bazel test //... has passed for a commit and its results are uploaded, the writer records that, as one more entry in the shared cache:

ac/sha256("inkylabs-verified:<commit sha>")    {"verified": "bazel test //...", "commit": …, "by": …, "at": …}

It is written last, after the upload, as an action result is written after its outputs, and it stands for everything: every target under //... built and every test passed for exactly that commit, whose tree pins every input (Bazel’s version, the nixpkgs commit, inkylabs_bazel’s commit). Reading it is one HTTPS request to https://bazel-cache.inkylabs.com/ac/<key>, with no Bazel, Nix or checkout. Bazel never asks for that key, since no step’s key is that hash. Only the cache’s writers (trusted workstations, through inkylabs-mr, and Inky Labs’ CI) can write it, so it is exactly as trustworthy as the cache. A test that passed once stays passed for that commit, flaky or not. Like any ac/ entry it expires after 60 days, and the commit is then simply built again.

Who reads it:

The review bot treats a verified commit with no pipeline as tested, and starts a pipeline only for one that isn’t verified (The review bot). A pipeline that ran for the commit counts either way: the marker vouches for bazel test //... and nothing else. So a push to a fork starts no pipeline (the rule below), unless it changes something the marker doesn’t cover: a project whose CI also plans infrastructure or builds an image starts one for a push that changes those paths, and only those (this handbook’s .gitlab-ci.yml is the example). A project that starts a pipeline on every push makes the bot wait for it, verified or not, for nothing. In a project that releases from tags, nor does a push to master: a merge is only of a commit that passed, fast-forward, so there is nothing left to build, and a release is its tag’s pipeline (Releases). A project that still deploys from master keeps that pipeline until it takes releases.
CI’s check reads it first and stops at once when it’s there, so a release’s pipeline, for the same commit the merge request built, takes seconds. When the job runs the tests itself and they pass, it writes the marker in turn.

workflow:
  rules:
    # A push to a fork starts no pipeline: the review bot starts one when the
    # commit isn't verified. Nor does a push to master, which the bot makes
    # with a commit that passed; it starts each release's pipeline itself.
    # (Only in a project that releases from tags, through ci/release.yml.)
    - if: $CI_PROJECT_NAMESPACE == "inkylabs/forks" && $CI_PIPELINE_SOURCE == "push"
      when: never
    - if: $CI_PROJECT_NAMESPACE == "inkylabs" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"
      when: never
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
    - if: $CI_COMMIT_BRANCH

check:
  extends: .inkylabs-bazel
  script:
    - |
      key=$(printf 'inkylabs-verified:%s' "$CI_COMMIT_SHA" | sha256sum | cut -d' ' -f1)
      if curl -fsSI -o /dev/null "https://bazel-cache.inkylabs.com/ac/$key"; then
        echo "verified: $CI_COMMIT_SHA already built and tested"; exit 0
      fi
    - bazel test //... --remote_download_minimal --test_output=errors
    - touch .inkylabs-tested
  after_script:
    - !reference [.inkylabs-bazel, after_script]
    - |
      # Only after the upload above, and only where CI may write the cache.
      [ -f .inkylabs-tested ] && [ "$CI_JOB_STATUS" = success ] || exit 0
      key=$(printf 'inkylabs-verified:%s' "$CI_COMMIT_SHA" | sha256sum | cut -d' ' -f1)
      printf '%s' "$INKYLABS_AWS_ID_TOKEN" > "$CI_BUILDS_DIR/.aws-token"
      printf '{"verified":"bazel test //...","commit":"%s","by":"%s"}' "$CI_COMMIT_SHA" "$CI_JOB_URL" |
        AWS_ROLE_ARN="$INKYLABS_BAZEL_CACHE_ROLE" AWS_WEB_IDENTITY_TOKEN_FILE="$CI_BUILDS_DIR/.aws-token" \
        AWS_ROLE_SESSION_NAME="gitlab-$CI_PROJECT_ID-$CI_JOB_ID" AWS_REGION=us-west-2 \
        aws s3 cp - "s3://inkylabs-bazel-cache/ac/$key" --only-show-errors || echo "verified: not recorded"
      rm -f "$CI_BUILDS_DIR/.aws-token"

The shared template (ci/bazel.yml) should carry this, so a project only extends it; it lives in each project for now. (A change to ci/ alone doesn’t rebuild the base image: it runs the example in the published :latest. Only a change to images/ci/ or bazel-cache-sync rebuilds it.) The template’s own after_script exits early where CI may not write the cache, which skips the marker too.

The base image marks every directory safe for git. It runs jobs as its own builder user, and the runner checks the repository out as another, so git would refuse the checkout (“detected dubious ownership in repository”) in any job that runs it (git rev-parse, git fetch, git checkout). The check guards a shared machine against a repository another user planted; a job’s container holds only its own checkout, so the image’s /etc/gitconfig trusts every directory (safe.directory = *), and the example job runs git rev-parse to prove it. Before SVRBC’s image did this (2026-10-08), each job that ran git needed git config --global --add safe.directory "$CI_PROJECT_DIR", and analytics.svrbc.org’s pages job lost a release (v0.2.0) for lack of it.

The CI image is a Bazel step

//images/ci:image builds the shared CI image: Nix and nix2container build it from images/ci/image.nix, from the same pinned nixpkgs as every tool, with no Dockerfile, no container runtime and no privileges. So it is built and tested like everything else: inkylabs-mr builds it and runs //images/ci:image_test (the user, the PATH, the tools, git’s trust, Nix’s settings, compressed layers) before a push, the shared cache keeps it, and the base-image job only uploads it (bazel run //images/ci:push), on the group’s runner. The archive is reproducible: an unchanged image is the same bytes.

Debian underneath, pinned by digest. An image of only Nix store paths can’t run Bazel: Bazel’s own binary, and toolchains it fetches prebuilt (its remote JDK), expect an ordinary Linux loader at /lib64/ld-linux-x86-64.so.2. So the image starts from debian:bookworm-slim and adds everything else from Nix, in /usr/local/bin, first on the job’s PATH and on the plain PATH Bazel gives an action. (Debian keeps /bin as a link to /usr/bin, so not there.)
Nix works inside, single-user, as builder, who owns /nix; its database lists every store path the image holds.
The CA bundle is where tools look by default, at Debian’s path and in nix.conf: a Bazel action starts with an empty environment, so SSL_CERT_FILE doesn’t reach it, and without the bundle Nix silently builds from source instead of downloading.
Layers are compressed (gzip): 273 MB to download, against 351 MB for the Dockerfile image it replaced at SVRBC (2026-10-08).

SVRBC’s image had been a Dockerfile until 2026-10-08, built with BuildKit on GitLab’s hosted runners until their minutes ran out, then with Kaniko. Not yet built for Inky Labs: there is no Inky Labs CI image or registry to push it to yet. To try a change by hand: bazel build //images/ci:image, then load it into Docker with skopeo (skopeo copy oci-archive:<image.tar> docker-daemon:<name>).

GitLab.com’s shared runners start every job from a clean state, so each job downloads its toolchain from the Nix binary cache. If that proves slow, the fix is to add the common tools to the image (from the same pins) or a Nix binary cache near the runners.

Making builds fast: practices for every project

Each of these came out of SVRBC converting its doreancon.org, and each applies to any Inky Labs project. They are as much part of the build practices as the rules at the top of this page.

Build everything once, before release

The pipeline that builds a merge request (in its fork) builds everything, the deployable site included, with bazel build //.... The merge is fast-forward, so the release’s pipeline, for that same commit, finds it all cached. Nothing heavy is ever built twice.

Don’t download what a job won’t read

bazel build downloads every top-level output by default, cache hit or not. A job that only needs to know everything is built passes --remote_download_minimal: cached steps are confirmed, not fetched. Then fetch only what the job actually uses (bazel run //:assets fetches the assets it installs; a deploy job builds just the target it ships). SVRBC’s doreancon.org’s cached check went from 148 s to 92 s; it had been downloading about 800 MB (node_modules and two site builds) it never read.

Make the release only deploy

A release tag exists only for a commit the review bot merged, which passed before it merged (Releases), so the release job neither rebuilds nor retests. It builds (that is, fetches) the one deployable target and deploys it.

npm dependencies are a build step

//:node_modules runs npm ci from package.json and package-lock.json and packs the result as a tarball, so every step that needs the dependencies depends on the lock file, not on whatever is installed on the machine:

skip only the root package’s own postinstall (npm ci --ignore-scripts) when it needs the whole app (as nuxt prepare does), then run the native modules’ install scripts that matter (npm rebuild better-sqlite3);
pack it with tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner, so an unchanged lock gives identical bytes and nothing downstream rebuilds.

Building a Nuxt (Vite) site inside Bazel

SVRBC’s doreancon.org’s scripts/bazel/nuxt-build.sh is the worked example. Three traps:

Vite follows symlinks, and in a Bazel sandbox every source is a symlink into the checkout. Built in place, the site would read the checkout’s own node_modules and generated files. Copy the declared inputs, dereferenced (cp -RL), into a directory of the action’s own, and build there.
Tools look upward for configuration. The sandbox’s own links to tsconfig.json and package.json sit one directory above that copy, and Vite’s TypeScript loader found the parent tsconfig.json and failed with “Tsconfig not found”. Remove the sandbox’s links to the config files once they are copied.
Run nuxt prepare first, since it’s the postinstall that the dependency step skipped; it writes the .nuxt/tsconfig files the build needs.

Work inside the action’s directory, never /tmp: a site plus its node_modules is a couple of gigabytes, and /tmp is often memory.

Deploying from the cache

The release job deploys the site CI already built: a Bazel step builds it with the host’s Nitro preset (aws-lambda), and the deploy uploads only what changed. Hosting has the setup and its traps.

Inputs and outputs

Keep generated, Git-ignored paths out of globs. A glob over public/** picks up whatever a developer last generated there. Exclude those paths and give the build Bazel’s own outputs instead.
A script built into Bazel keeps working outside it. Give it overrides for where it writes (SVRBC’s doreancon.org has HYMNAL_OUT, QR_OUT, BULLETIN_OUT), and find the repository from its own path without following symlinks (node --preserve-symlinks-main; Python’s Path(__file__).absolute(), not .resolve()).
A script that edits files in place gets a copy of what it edits, made into its own outputs first.
Outputs Bazel can’t list ahead of time (one QR code per project in a JSON file) go into a tarball.

Large files go to an assets domain

A site’s large files (PDFs, audio, video) don’t belong in its deploy: every release fetches them from the cache and uploads them to the host again, and the host’s bandwidth pays for every download. They go to the site’s assets domain instead, assets.<domain> (see Domains):

Content-addressed keys. A Bazel step hashes each file and maps the path the site links to onto <sha256>/<name> in the site’s bucket. A key names exact bytes, so it’s immutable (cached for a year) and a changed file is a new key, never an overwrite.
The site keeps its paths and redirects them (Nuxt routeRules, 302), so no link changes and npm run dev still serves the files locally.
Upload before deploy, only what’s new. The release job lists the bucket, uploads the keys it lacks with a role that trusts only that project’s release tags, and only then deploys the site that points at them. It fetches those files from the build cache only when one is new.
CORS. Anything the site fetches from there (on SVRBC’s doreancon.org, its countdown music) needs Access-Control-Allow-Origin: CloudFront’s managed SimpleCORS policy.

SVRBC’s doreancon.org moved 96 MB (19 files) there, halving its deploy from 204 MB to 105 MB. See SVRBC’s infra/doreancon-org-assets/.

Prove a new build equal before switching

Before a Bazel step replaces an old build path, compare their outputs: bytes where the tool is deterministic (MIDI, manifests, JPEG renders), text and page counts and then pixels where it isn’t (PDFs), and structure where hashes legitimately differ (a site’s routes and file list). SVRBC’s doreancon.org found two real problems this way, a host font leak and a missing Vercel route, before either reached production.

Render both in the same browser on the same machine. Against files built elsewhere (CI’s pinned fonts, another Chrome), every page differs a little, and a real difference hides in the noise: SVRBC’s doreancon.org’s bulletin icons printing as solid squares did, until production and the new build were rendered side by side locally. For a site, site-compare does this page by page: screenshots at each width and color scheme, and the text. SVRBC checked doreancon.org’s move to Astro route by route this way.

Why this setup

The goal: never redo a build step whose inputs haven’t changed, whether on a workstation, in CI, or on another machine, while keeping local edit-and-rebuild fast.

Why not Nix on its own? Nix caches whole derivations and decides whether to rebuild by inputs. It has no early stop on unchanged output (that needs content-addressed derivations, which are experimental), and every build pays for evaluation and copying the source into the store. That’s fine for CI and slow for an edit-and-see loop. We still use Nix, for what it does best: pinned, reproducible tools.
Why not a Docker image per project? Images drift from the build’s idea of its inputs. Upgrade Chromium in an image, forget to tell the build, and the cache serves stale artifacts. They also add a pipeline stage and tag juggling to every repo.
Why no cache server? Bazel’s remote cache speaks gRPC, or HTTP GET and PUT. S3 needs signed requests, so Bazel can’t write to it. But it can read from it, and writes don’t have to go through Bazel: a trusted machine uploads its disk cache afterwards. That removes the server (bazel-remote on EC2, with its patching, TLS and password file) and leaves a bucket, a CDN and an upload script. The cost: new results appear when a job finishes, not as each action completes.
Why not hand-rolled input hashes? SVRBC’s doreancon.org had one, build-stamps.json, which skipped unchanged hymns. But it only hashed the inputs someone remembered to list (not tool versions), and it couldn’t share results between machines.

Pilot results

These are SVRBC’s (Pilot results), from its doreancon.org; Inky Labs hasn’t piloted anything of its own yet. They depend on Bazel, Nix and GitLab, not on whose project it was.

doreancon.org’s hymn engraving, measured 2026-10-01 on the bazel-pilot branch. There are 18 hymns. MuseScore 4.7.4, Xvfb and Node 24 come from one pinned nixpkgs commit through rules_nixpkgs_core 0.14.0, with Bazel 9.2.0.

Case	Result
Engrave all 18 hymns, warm toolchain	12.6 s
Edit one hymn	1.3–1.4 s, one action (1.8 s before the fonts were pinned)
Revert that edit	0.09 s, from the disk cache
Change a manifest field the engraver doesn’t read	0.23 s: the per-hymn cuts re-run, all engravings skipped
Fresh checkout, shared cache	36/36 cached; 12.7 s, all of it Bazel start-up
Cold container (base image, nothing cached)	60 s, about 55 s of it toolchain download
Cold container, remote cache warmed by trusted CI	36/36 remote hits, 0.46 s critical path, 42 s total
Same hymn engraved directly, warm MuseScore, no Bazel	0.8 s
Fresh output base, reading `bazel-cache.svrbc.org`	36/36 remote hits, no warnings
`examples/hello` on GitLab.com, second `master` pipeline	1/1 remote hit, uploaded by the first through OIDC
The whole hymnal (all five steps: 92 actions), cold	15.7 s locally
The whole hymnal on GitLab.com, after one upload	92/92 remote hits; 57 s, nearly all toolchain download
doreancon.org release v1.1.31 (2026-10-03): every asset, tests, Vercel deploy	279 s, 94/94 Bazel steps cached; the Docker-image deploys it replaced took 248–265 s
doreancon.org check, everything cached, the whole site in Bazel (2026-10-04)	148 s; 92 s with `--remote_download_minimal`

What SVRBC learned:

A single local edit is not faster. It’s about 0.5 s slower than running the engraver directly (Bazel’s sandbox and start-up). The gains are elsewhere: no work is repeated across machines, nothing re-runs when an output comes out unchanged, and the tool versions are part of every cache key.
Nix’s fontconfig reads the host’s fonts. On a non-NixOS machine it falls back to /etc/fonts and scans /usr/share/fonts, so text could set differently per machine with nothing in the cache key to show it. Pin a fonts.conf that lists only Nix fonts and a prebuilt cache, and pass it as FONTCONFIG_FILE (doreancon.org’s nix/fonts.nix). That fixed hermeticity and also removed most of the per-action cost: a fresh HOME no longer rescans fonts.
Bazel reproduces production exactly. The whole hymnal through Bazel matched the npm pipeline run in doreancon.org’s build-tools image (the one production deploys from): both manifests, all 18 MIDI files and all 18 crops, and all 374 tests passed against it. Since 2026-10-02 the deploy ships Bazel’s hymnal.
Scripts that edit files in place (annotate, the manifest step) work as Bazel actions by copying what they edit into their own outputs first.
Python’s Path(__file__).resolve() escapes the sandbox, as Node’s symlink resolution does: it follows the symlink into the source tree. Use .absolute().
Once everything cacheable is cached, a release costs what isn’t. Of v1.1.31’s 279 s, none went to building a cached step. About 60 s went to Bazel fetching its pinned tools (MuseScore, Chromium) even though every step was a hit, about 120 s to Vercel (pull, the Nuxt build, upload), 26 s to the bulletin PDFs, and the rest to starting the job and npm. Faster releases now mean a warm image generated from the Nix pins, and the Nuxt build as a Bazel action.
MuseScore’s output isn’t byte-stable, so nothing downstream of a fresh engraving can be skipped. The skip happens before MuseScore, at the per-hymn manifest cut.
xvfb-run -a races when actions run in parallel: most X servers die on a shared display number. Start Xvfb -displayfd directly instead.
GTK starts gvfs daemons that leave a FUSE mount in $HOME. Set GIO_USE_VFS=local.
Bazel falls back to processwrapper-sandbox on Ubuntu workstations (AppArmor restricts unprivileged user namespaces) and in unprivileged containers. It works, but it can’t hide absolute paths outside the sandbox.

Open questions

SVRBC’s doreancon.org pilot answered these before its other repositories converted. The answers carry over, but Inky Labs has to check the last of them on its own infrastructure: