Skip to content

fix(devcontainer): mount the cache volumes at the roots the tools use - #231

Closed
Syndic wants to merge 2 commits into
mainfrom
claude/devcontainer-cache-persistence
Closed

fix(devcontainer): mount the cache volumes at the roots the tools use#231
Syndic wants to merge 2 commits into
mainfrom
claude/devcontainer-cache-persistence

Conversation

@Syndic

@Syndic Syndic commented Aug 16, 2026

Copy link
Copy Markdown
Owner

The gap

Both named volumes missed most of what they were supposed to persist. Measured inside a running container on this branch's base:

Cache Real path Persisted before? Size
Bazel output base ~/.cache/bazel yes 23G
GOCACHE ~/.cache/go-build no 967M
bazelisk downloads ~/.cache/bazelisk no 61M
pre-commit hook envs ~/.cache/pre-commit no 13M
uv ~/.cache/uv no
$GOPATH/pkg (module + checksum-db caches) /go/pkg no 530M
~/go yes empty

Two separate bugs:

  • ud-bazel-cache targeted ~/.cache/bazel, one directory inside the cache root, so its four siblings were container-local.
  • ud-go-cache targeted ~/go, but features/go bakes GOPATH=/go into the image ENV (docker inspect … | grep GOPATHGOPATH=/go). ~/go does not exist in the image at all and nothing in the container writes it.

Neither fails loudly. The container just rebuilds those caches every time, which reads as "devcontainers are slow".

The fix

Two volumes, each at the root of what it persists:

  • ud-cache~/.cache
  • ud-go-pkg-cache/go/pkg

Why the whole of ~/.cache, not per-tool volumes. Per-tool only persists what someone remembered to enumerate, which is precisely the failure being fixed — bazelisk was silently missing and nobody noticed. Mounting the root means a tool that starts caching under XDG is covered with no edit. Nothing under it wants to stay ephemeral either: every entry is content-addressed or key-validated by its own tool, and CI builds cold, so a stale local cache can't reach main.

Why $GOPATH/pkg and not $GOPATH. /go/bin is image content — features/go builds ten tools there at image-build time — and Docker only seeds a volume from the image while the volume is empty. A volume over all of /go would freeze bin/ at whatever the image held on first mount, so a later feature bump would install tools nobody ever sees. Persisting it buys nothing anyway: post-create.sh reinstalls its six pinned tools over the image's copies on every create. pkg/ is the derived half of GOPATH, bin/ is the artifact half.

Why not $GOMODCACHE alone. Tried it; it fails outright. The checksum-db cache is a sibling of the module cache at /go/pkg/sumdb, and Docker creates the /go/pkg mountpoint parent root-owned, so the first go install dies on open /go/pkg/sumdb/sum.golang.org/latest: no such file or directory.

Nesting a ~/.cache volume under the existing ~/.cache/bazel one would have worked mechanically, but leaves two volumes with overlapping meaning and no axis separating them — bazel clean --expunge already gives per-tool discard from inside the container. So ud-bazel-cache is retired rather than kept.

post-create.sh's chown follows the new targets and is now guarded on current ownership. Unconditional chown -R over a warm ~/.cache is minutes spent re-asserting ownership that is already correct, and it would flatten the go feature's vscode:golang group.

Correction to an earlier claim in this PR

An earlier revision described ~/go as having "persisted an empty directory since it was added". A reviewer observed 731M and ten binaries at ~/go in a different worktree's container on the same host, which looked like a direct contradiction. Reconciled — both observations are correct and neither undermines the change:

  • The reviewer's container still runs the old config, which mounts ud-go-cache at ~/go. This branch's containers mounted that same volume at /go, where post-create.sh filled it with 540M of module cache. A volume's content is shared across containers; its mount point is per-container. So what reads as "~/go has been doing real work" is this branch's own output arriving through the shared volume.
  • The Aug 12 mtimes cited as pre-recreate survivors are image-layer dates, not volume survivors. Proven by running the image with no volumes at all:
$ docker run --rm --entrypoint /bin/bash vsc-devcontainer-cache-persistence-… -lc 'ls -la /go/bin; ls -la /home/vscode/go'
-rwxrwxr-x 1 vscode golang  6434527 Aug 12 19:47 golint
-rwxrwxr-x 1 vscode golang  8463572 Aug 12 19:47 goplay
-rwxrwxr-x 1 vscode golang 13705373 Aug 12 19:47 revive
-rwxrwxr-x 1 vscode golang 15228660 Aug 12 19:47 staticcheck
…                                                                  (ten total)
ls: cannot access '/home/vscode/go': No such file or directory

golint, goplay, revive and staticcheck are exactly the four of the ten that post-create.sh does not install, which is why they keep the image's date while the other six carry the recreate's. They are re-provided by the image on every rebuild — nothing is lost by not persisting them. ~/go is genuinely vestigial, and this PR drops the mount rather than orphaning anything.

The reviewer's GOPATH=/go / GOMODCACHE=/go/pkg/mod finding is confirmed, and the pushback is what surfaced both the /go/bin shadowing and the sumdb breakage above.

Validation

Ran in the devcontainer. Recreated with devcontainer up --remove-existing-container; sentinel files written in the previous container survived in both volumes, and /go/bin is demonstrably not shadowed:

=== sentinels written in the PREVIOUS container ===
2026-08-16T11:10:36Z     # /go/pkg/SENTINEL
2026-08-16T11:10:36Z     # ~/.cache/SENTINEL
=== survived ===
530M  /go/pkg/mod          12K  /go/pkg/sumdb
977M  ~/.cache/go-build    13M  ~/.cache/pre-commit
=== /go/bin: image dates intact next to post-create's ===
-rwxr-xr-x  golangci-lint  Aug 16 04:10     # post-create, pinned
-rwxrwxr-x  staticcheck    Aug 12 12:47     # image layer, unshadowed
=== ownership (go feature's group preserved on /go) ===
drwxrwsr-x vscode golang /go       drwxr-xr-x vscode vscode /go/pkg

Warm-vs-cold, same commands:

cold immediately after recreate
go build ./... 3.5s 0.52s
pre-commit install-hooks 4.4s 0.36s
postCreateCommand redownloads bazel 9.2.0 15s, no download

bazel test //... — 21 tests pass. pre-commit run --all-files — all hooks pass except a pre-existing SC1091 in meta/devcontainer-base/scripts/devcontainer-plumbing.sh, untouched here.

New tests are mutation-checked: widening the mount to /go, and dropping a target from post-create's chown loop, each make //.devcontainer:test_devcontainer_config fail.

Notes for the reviewer

  • After merging, docker volume rm ud-bazel-cache ud-go-cache reclaims the two now-unreferenced volumes. Other worktrees' running containers still hold the old mounts until they are recreated.
  • .claude/CLAUDE.md gains a "Devcontainer cache volumes" section — expect a possible trivial conflict with ci(bazel): namespace the repository cache per workflow #229, which also appends to that file.

🤖 Generated with Claude Code

The two named volumes each missed most of what they were meant to persist.

`ud-go-cache` targeted `~/go`, but `features/go` exports `GOPATH=/go`, so the
volume held an empty directory and GOMODCACHE (`/go/pkg/mod`, 530M) plus the six
`go install`ed tools in `/go/bin` were discarded on every rebuild.

`ud-bazel-cache` targeted `~/.cache/bazel`, one directory inside the cache root,
leaving its siblings ephemeral: `go-build` (967M), `bazelisk` (61M, a fresh
bazel download per rebuild), `pre-commit` (13M) and `uv`.

Mount the roots instead — `ud-cache` at `~/.cache`, `ud-go-cache` at `/go` — so
a tool that starts caching under XDG is covered without an edit here. Nothing
under either root needs to stay ephemeral: every entry is content-addressed or
key-validated by its own tool, and CI builds cold.

post-create.sh's chown follows the new targets and is now guarded on current
ownership, since recursing tens of GB to re-assert correct ownership on every
rebuild is minutes for nothing. The guard also preserves the go feature's
`vscode:golang` group on `/go`, which an unconditional chown would flatten.

test_devcontainer_config.py asserts both couplings the change creates: the
volume targets are exactly the two cache roots, and post-create.sh chowns
exactly that set (a target with no entry mounts root-owned and unwritable).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.45%. Comparing base (5e731a7) to head (e5e53a6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #231      +/-   ##
==========================================
+ Coverage   84.07%   84.45%   +0.37%     
==========================================
  Files          39       40       +1     
  Lines        2129     2193      +64     
  Branches       91       99       +8     
==========================================
+ Hits         1790     1852      +62     
- Misses        324      325       +1     
- Partials       15       16       +1     
Components Coverage Δ
Go 86.14% <ø> (ø)
Python 79.79% <96.92%> (+2.10%) ⬆️
Category: apps ∅ <ø> (∅)
Category: infra ∅ <ø> (∅)
Category: libs ∅ <ø> (∅)
Category: meta 79.79% <96.92%> (+2.10%) ⬆️
Category: services ∅ <ø> (∅)
Category: tools 86.14% <ø> (ø)
Project: meta/scripts 79.79% <96.92%> (+2.10%) ⬆️
Project: tools/network_infrastructure_maintenance 86.14% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 011a9aa...e5e53a6. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Review pushback on the previous commit's `/go` mount prompted a closer look at
what actually lives under GOPATH, and the mount was wrong in two ways.

`/go/bin` is image content: `features/go` builds ten tools there at image-build
time (`/usr/local/etc/vscode-dev-containers/go.log`), of which post-create.sh
reinstalls six at pinned versions. Docker seeds a named volume from the image
only while the volume is empty, so a volume over `/go` freezes `bin/` at
whatever the image held on first mount — a later feature bump would install
tools nobody ever sees. Persisting it buys nothing either, since post-create
rewrites its six on every create regardless.

`/go/pkg/mod` alone is too narrow the other way: the checksum-db cache is a
sibling at `/go/pkg/sumdb`, and Docker creates the `/go/pkg` mountpoint parent
root-owned, so the first `go install` fails on

  open /go/pkg/sumdb/sum.golang.org/latest: no such file or directory

`$GOPATH/pkg` is the boundary that holds: `pkg/` is the derived half of GOPATH
(mod + sumdb, absent from the image — the feature purges it), `bin/` is the
artifact half. Verified after recreate: `/go/bin` keeps its Aug 12 image dates
for the four tools post-create doesn't install, while `mod` and `sumdb` both
persist across `--remove-existing-container`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Syndic

Syndic commented Aug 19, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #234

@Syndic Syndic closed this Aug 19, 2026
@Syndic
Syndic deleted the claude/devcontainer-cache-persistence branch August 19, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant