Zed and Deno were my Rust benchmark builds, and they showed me what it costs to move large target directories between runners. A Rust target/ directory can hold tens of gigabytes of useful work. Reusing it can skip a lot of compilation, but moving it between fresh runners can get expensive enough to eat much of that saving. I wanted something closer to coming back to a checkout that already has a warm target directory: use the existing runners, restore the useful build state, and let Cargo decide what still needs to run. A faster machine still helps when there's a lot of compilation left. What interested me was how much work I could avoid on the machine already running the job.
Keeping useful target state
I wrote about the three parts of Rust build caching in an earlier post: Cargo's downloaded dependencies, its target directory and sccache each keep different work. This post is about the target directory, whose transfer time can eat much of the time caching saves. One reasonable answer is to save less of it. Swatinem/rust-cache lets you choose which crates to keep, and leaves workspace crates out by default. That makes the archive cheaper to move, on the expectation that the workspace will rebuild.
The trade-off depends on the project. A workspace can contain expensive outputs that stay useful across many runs. Leaving them out makes the archive smaller, but the next build may spend longer recreating them than it would have spent restoring them. I wanted to cut the time and resources it takes to save and restore that work instead. A large target directory doesn't have to mean an equally large upload after every build, because much of it can be identical to the previous run even when some files have changed.
Why I moved to content-defined chunks
Splitting an archive into fixed-size pieces looks like a simple way to reuse it: hash each piece, upload the missing ones and keep references to the rest. The problem is where the boundaries fall. Insert a few bytes near the start of the archive and everything after shifts relative to those fixed offsets, so later pieces get different hashes even though most of the files underneath haven't changed. That's what happened in a local Cargo experiment in August. A one-line rebuild changed a 24.7 GB target directory, and the fixed 256 MiB version reused none of its 93 chunks. Dropping the chunk size to 8 MiB didn't help: none of those 2,948 chunks matched either. An early changed fingerprint entry had shifted the stream.
FastCDC picks boundaries from the content, so after a changed region it finds matching boundaries again and reuse recovers further through the archive. In that experiment it reused 96.9% of the decoded bytes. The new compressed chunks came to 97.6 MB, against 3.58 GB for the complete compressed successor. I first described that experiment in the BuildKit retrospective. The target was still 24.7 GB when reconstructed; 97.6 MB was the new compressed data to transfer. Reusing existing chunks means you can keep a large directory without sending all of it again.
The work around the chunker
BoringCache uses object storage, so I've spent a lot of time on what the client does before and after each request. For a changed directory, the CLI produces a deterministic tar stream, finds content-defined chunks in it, hashes them and checks them against verified descriptors from the previous snapshot. When a matching compressed chunk is still available remotely, the client can reuse it without compressing it again or writing another temporary copy. It compresses only the new chunks and uploads the missing objects. That saves local CPU and disk work too. Skipping the upload wouldn't help much if the client still compressed and wrote the same gigabytes on every run.
I've also worked on overlapping the stages of archive preparation, keeping concurrent work to what the machine can sustain, and avoiding unnecessary copies. More workers only help while they improve throughput; they also use memory and compete for disk access. The client can also skip reading an unchanged directory: when it can verify the previous snapshot and reliably tell that the selected directory didn't change, it reuses the snapshot without reading everything again. If it can't prove that, it reads and verifies the directory normally.
Restore has its own costs. The client downloads chunks concurrently, verifies and decompresses them and rebuilds the archive in order. A fresh runner still has to restore the selected target snapshot before Cargo uses it, even when the previous runner uploaded little new data. The Cargo adapter also keeps the source and artifact freshness relationships Cargo expects, because copying a directory quickly is no use if the next build treats all of it as stale. Cargo still decides which units need to rebuild.
What a 40 GB target looks like in practice
The Zed benchmark is the long-running case. Its Cargo target directory starts at about 15 GB after the first builds and grows to 43 GB as commits accumulate, and every rolling build restores it onto a fresh runner before Cargo starts. On Depot's 4 vCPU runners the restore took a median of 47 s over 66 builds, and a 43 GB target restored in 49 s. On GitHub's 4 vCPU runners the median was 87 s, and restores of more than 40 GB took about two to four minutes, depending on the CPU model.
| Cache | Time |
|---|---|
| BoringCache + | 12 min 57 s |
| Depot runner cache | 19 min 35 s |
- Same CPU:
- BoringCache was faster on all 30 builds, including the 29 where both caches ran on the same CPU model
- Release bundle:
- Zed's Linux release, built with its bundle script
Keeping the target is only worth it if the next build gets there sooner. In the Zed release bundle, both caches restore the Cargo registry and the target directory, and the BoringCache build finished 6 min 38 s sooner on average. Cargo decides what to rebuild from the restored outputs, so the saving comes from the work it didn't repeat, not from the transfer itself.
Target directories can grow despite small uploads
Reusing most chunks keeps uploads small, but the target directory can still grow. Cargo can leave older outputs behind as dependencies and build configurations change. If the next runner restores them, adds its new work and saves the result, old files carry on from snapshot to snapshot. Content-defined chunking makes them cheap to keep remotely, but they still cost disk space and restore time on every fresh runner.
In an August 25 Deno benchmark, the runner restored a 31.94 GB target in 115.2 seconds. After two build phases, the saved target had grown to 35.43 GB. The final save reused 5,932 of 6,248 chunks, about 95% by count, and still took 91.4 seconds. Most chunks were reused, and the target kept growing. Counting the chunks I didn't upload couldn't tell me whether all the files in the next snapshot were still worth keeping.
BoringCache restores and updates the configured target tag, with the existing platform and Git suffixes. Compiler, dependency and workspace changes don't add hash suffixes or pick a new tag; Cargo decides which restored outputs need rebuilding. Dependency archives and sccache stay available separately. When I added that target reuse in 1.30.2, it didn't prune unused artifacts or set a size budget. BoringCache 1.40.0, released on 6 October, added target pruning with a default 40 GiB budget. Chunk reuse cuts repeated uploads, and pruning deals with the older outputs that would otherwise keep piling up in the target.
Knowing what Cargo still uses
Pruning needs a reliable answer to when an artifact was last used. Its content hash can't tell you that, and neither can the fact that I uploaded it recently: I may just have copied an old file into another snapshot. So I use Cargo's JSON build messages to record which artifact groups a command used. After a successful observed command, BoringCache can remove older, unused groups to work towards the budget. It keeps active outputs and files it can't safely classify, even if that leaves the target over budget. If the command doesn't provide complete usage information, BoringCache keeps the files.
This uses Cargo's build output without depending on the nightly mtime-on-use experiment. Since BoringCache 1.40.2, eligible wrapped commands with non-terminal output can supply that information without an explicit message-format flag. Set target-budget under [adapters.cargo] to change the budget, or set it to "0" to turn pruning off. The Cargo adapter guide covers observation, protected outputs and commands that can't be pruned. Cargo's automatic cleanup of its home-directory cache still handles downloaded dependency data separately. A smaller archive only helps if it cuts the time and resources the next build needs, and removing an expensive output that gets rebuilt straight away makes both worse.
I'm still working on letting a fresh runner carry on from useful work already done. Matching a persistent builder means accounting for the restore, the remaining compilation, the save and the state that builds up over time, and I'd need a direct comparison on the same workload and hardware before claiming the same times. I want to make that reuse cheaper locally and more selective over time. If you try the Cargo adapter, measure the complete job and watch the target across several source changes. Those are the numbers I'm trying to improve.