Skip to content
  • BuildKit
  • Benchmarks

The BuildKit backend we deleted after it worked

I tried moving BuildKit state between ephemeral runners. Restoring it was fast, but changed builds were slower than the simpler cache backend.

Gaurav Tiwari Updated 15 min read

I didn't want to fork BuildKit. BoringCache already had a CLI that could walk an archive, split it into content-addressed pieces, upload what was missing and restore it on another machine, and I thought I could reuse most of that code without touching BuildKit. I wanted a fresh GitHub Actions runner to restore the BuildKit state the previous runner had saved.

Persistent runners

Most GitHub-hosted jobs start on a new virtual machine. When the job finishes, the local BuildKit state goes with it. The next job can import an external cache, but it can't see the previous builder. A persistent runner avoids this: BuildKit keeps its state on disk between builds, so there's nothing to export and import, and it just reads the state it wrote. Its cache mounts also stay available between builds unless garbage collection removes them.

But that state stays with the builder. If I build locally, move the job to another CI provider, use an ephemeral runner or deploy from another machine, the new builder can't access the previous builder's local state. I wanted to save enough of the BuildKit state after one job to restore it on a fresh builder for the next one.

The first subset of BuildKit state

I used PostHog because it has a large, real Docker build rather than a small cache fixture. In the first useful run, in July, the BuildKit root was 31.38 GB. About 26.69 GB was expanded snapshot data, and the remaining 4.69 GB was the state I tried to move.

PostHog · first state-sync experiment
BuildKit root             31.38 GB
snapshot directories      26.69 GB
state without snapshots    4.69 GB

save                          34s
restore to fresh builder      19s
same-source build              20s  · 68 cached steps
changed successor          failed  · missing parent snapshot bucket

The same-source build made me think at first that the saved state was enough.

The local PostHog test took 588.8 seconds to restore, 225.3 seconds to load BuildKit and 2.6 seconds to build: 816.7 seconds total versus 485.6 seconds cold.
In the later local PostHog experiment, the complete restored build took 816.7 seconds, compared with 485.6 seconds for a cold build.

The first changed build failed

The next source change broke it. I had left out the snapshot directories because they looked like they could be rebuilt, but the metadata still had parent relationships pointing at them. The same source could reuse what was already there, but a changed successor needed a parent snapshot I'd excluded, and BuildKit correctly refused to continue. That showed me the difference between an archive and BuildKit state. A directory archive records which files it contains. BuildKit state is also a graph of cache records, results, parents, snapshots, leases and immutable content. A generic filesystem tool can copy those files, but it doesn't know which records make a valid portable cache.

Restoring the state in 19 seconds didn't prove that the restored builder could build changed source correctly. From then on, I tested three things: restore on a fresh builder, rebuild the same source, and then build a changed successor.

The backend grew

I kept going. In the next design, BuildKit decided which records were eligible, each generation was published atomically, and a failed build left the previous head in place. Metadata arrived first, and immutable layer bodies could stay remote until BuildKit asked for them. I also tried preparing the state during the build: as immutable refs appeared, the backend prepared them while later Dockerfile steps were still running, with a governor that backed off when the build needed CPU or I/O and used more of the machine once only cache work remained. By then it was no longer a directory backup. It had lineage checks, signed generations, lazy range reads, cleanup rules, retry boundaries and a fresh-runner canary. And it worked.

Larger runners

I also wanted to know how much a larger machine would help. The PostHog test ran the exact upstream Dockerfile on 4, 8 and 16-core AMD64 runners:

PostHog BuildKit state performance by runner size
Runner Build Total job State overhead Published state
AMD64 · 4 cores 816s 848s 109.9s 6.916 GB
AMD64 · 8 cores 530s 574s 64.2s 6.917 GB
AMD64 · 16 cores 339s 378s 35.6s 6.917 GB

The larger runners helped, but four times the cores didn't make the build four times faster: 816 seconds became 339. The state overhead fell from 109.9 seconds to 35.6, but it didn't go away. The full numbers, including the ARM64 lane and exact source commits, are in the PostHog benchmark pull request.

The benchmark caught stale output

An earlier version combined the state backend with a Turbo remote cache inside the PostHog build. Changed commits added files under frontend/public/services, but PostHog's Turbo build inputs didn't include frontend/public, so a remote hit replayed the older frontend output. The cached lanes kept reporting 6,103 collected static files while an uncached lane produced 6,108 and then 6,110. The benchmark looked faster because one of its caches had skipped work it should have done. I turned Turbo off in the main benchmark and kept it only for diagnosing the issue, so the state results used the upstream Dockerfile unchanged and included the frontend work the bad cache hit had skipped. A cache hit that returns stale output is a bug, however fast it is.

Changed builds were still slower

The final packs-based design could restore a generation lazily and produce 18 and 16-second same-source builds, with 76 cached vertices. This time the cache graph was valid, and a fresh builder could use it. Changed builds were the problem. A small source change could still make the state path process two to six gigabytes locally, and in the measured PostHog sequence it recreated about 2.1 GB of pack bodies per realistic changed commit, even when only around 150 MB was new to the remote content store.

PostHog packs-based state backend time by build phase
Phase Total build State processing State work after build
Cold PostHog 610s 30.7s about 23s
Large endpoint change 492s 20.4s about 22s
Same source 18s / 16s small about 2.84s

The roughly 500-second number was the whole changed build, not a 500-second state save. Much of the state work ran beside the build, so looking only at the final save would miss the CPU, filesystem reads, compression, range fetches and local writes already done during the solve. The final PostHog state run, also in July, recorded both the same-source hits and the slower changed builds. In the paired builds, realistic changed state builds averaged about 513.7 seconds, against about 466.0 seconds for the simpler layer exporter. The backend was correct and fast when the source was unchanged, but on changed builds it was still doing more work than the simpler exporter.

Would FastCDC help now?

It might reduce the bytes, but I don't know whether it would change the result. BoringCache archives now use FastCDC boundaries over a canonical tar stream. In one local Cargo experiment, a one-line rebuild changed a 24.7 GB target directory. Fixed chunks reused none of the shifted stream, while FastCDC realigned after the changed regions, reused 96.9% of the decoded content and cut the compressed successor from 3.58 GB to 97.6 MB. That's a lot less compression and upload work, but the client still has to walk the directory and produce one canonical tar stream before it can find those reusable chunks.

BuildKit state adds another problem. It isn't just a stable directory archive, and raw OCI or BuildKit layer tar streams can contain representation changes, such as fresh timestamps, that change chunk boundaries or content. A fresh builder still needs the valid graph rebuilt, and making all of that lazy would bring back the range reader, virtual content provider and generation lifecycle that made the deleted backend complicated.

I tried it again

I did end up running the experiment again. I still wanted to test whole-state caching, mostly because BoringCache has FastCDC now, and it's much better at finding the parts of a large archive that haven't really changed. The Cargo result made me wonder whether better chunk reuse would help the state backend too. Maybe the old backend had been doing too much work simply because it couldn't see that most of the bytes were still the same.

I didn't bring the old backend back. Instead I stopped a normal BuildKit builder after the PostHog build, took its complete state exactly as it was on disk, sent it through the archive system BoringCache already has, and restored it onto a fresh builder. I didn't change the Dockerfile or add another state format inside BuildKit. I just wanted to see whether saving and restoring the complete state could get close to the reuse and build times of a persistent builder.

Some of it looked promising. The BuildKit root was 35.92 GB across 885,800 filesystem entries. Its metadata-preserving archive was 36.91 GB, which FastCDC and compression cut to 9.88 GB of unique remote content. Once the archive existed, creating its chunk graph took 16.3 seconds, and an exact repeat save finished in 15.4 seconds without uploading the same chunks again. The restored builder was correct too: all 74 Dockerfile steps were cached, the final image digest was identical, and the actual build took 2.6 seconds.

But creating and restoring the archive took far longer than the 2.6-second build. Creating it from the stopped BuildKit volume took 606.9 seconds. Restoring it from BoringCache took 588.8 seconds, and putting the reconstructed state into a fresh BuildKit volume took another 225.3 seconds. The complete restored build took 816.7 seconds, and building PostHog cold took 485.6. The cache worked, but waiting for it was slower than doing the work again.

I also tried skipping the intermediate archive file and streaming the chunks straight into the fresh BuildKit volume. That saves the extra 36.91 GB file, but it doesn't remove the work of recreating nearly 886,000 entries, directories, whiteouts and overlay attributes. I stopped that local test once it had run too long for the restored build to beat a cold build. It was a deliberately simple implementation, so I don't count its unfinished time as a benchmark; the completed restore had already shown that this approach took longer than a cold build.

I thought about the other ways of moving the state too: copying filesystem blocks, taking a volume snapshot, keeping the data behind a lazy reader, or attaching the same disk somewhere else. Each one changes which data has to be transferred and rebuilt. Send BuildKit's portable blobs, and BuildKit has to create them and later turn them back into local snapshots. Send the snapshots as they are, and you have to move and recreate a much larger filesystem. Keep the same disk attached so neither happens, and you've built a persistent builder.

I realised these approaches mostly change when and where state is prepared and rebuilt. FastCDC can make the rolling save smaller, and the exact repeat showed it wouldn't upload unchanged chunks again. I didn't get as far as a changed build, so I still don't have a save time for a realistic source change, but restore alone had already taken longer than the cold build, and a faster save wouldn't change that. The experiment answered my question about whole-state restore, and it didn't change my decision to remove the state backend.

What I kept

The current BoringCache BuildKit backend keeps BuildKit's normal cache graph and works with its portable immutable layer descriptors. It asks the destination whether an exact body already exists before opening or materialising it, and it prepares missing bodies during the solve when the machine has spare CPU or I/O. It removes most of the export work at the end without trying to recreate another machine's whole builder on a fresh runner. I wrote about it in Docker build done. Still exporting cache…, and its current PostHog results are on the benchmarks page.

A persistent runner still has an advantage, because keeping local state avoids any translation, but that state stays attached to the runner. BoringCache stores portable layer content so the cache can be used locally, in CI and on another machine. I needed a smaller portable unit, not every byte BuildKit kept on disk.

Why I deleted it

I liked the state backend. It had taken a lot of work and, by the end, it restored fresh builders, survived failed publications, loaded bodies lazily and handled the same-source build correctly. But I had built it to save build time, so correctness alone wasn't enough. On the changed builds that mattered, the simpler backend was faster and much easier to run. So I deleted the working backend.

I think this is a normal part of building software. You build something, measure it properly, and sometimes find you learned a lot from it but shouldn't keep it. Three things from this one shaped the next backend: what BuildKit keeps on disk is much bigger than what needs to move between machines, a same-source hit doesn't prove a cache is correct, and cache work that moves into the build still has to be counted. I kept the backend that reuses portable layer content without restoring the full builder state.