Skip to content

fix(devcontainer): mount the cache volumes at the roots the tools use - #234

Merged
Syndic merged 3 commits into
mainfrom
claude/devcontainer-cache-roots
Aug 19, 2026
Merged

fix(devcontainer): mount the cache volumes at the roots the tools use#234
Syndic merged 3 commits into
mainfrom
claude/devcontainer-cache-roots

Conversation

@Syndic

@Syndic Syndic commented Aug 19, 2026

Copy link
Copy Markdown
Owner

The gap

Both named volumes missed most of what they were supposed to persist. Measured in a container built from current main (3b10601), with findmnt naming which paths were actually volumes:

Cache Real path Persisted before? Size
Bazel output base ~/.cache/bazel yes 23G
GOCACHE ~/.cache/go-build no 1.9G
bazelisk downloads ~/.cache/bazelisk no 61M
pre-commit hook envs ~/.cache/pre-commit no 13M
uv ~/.cache/uv no
$GOPATH/pkg (module + checksum-db caches) /go/pkg no 544M
~/go yes written by nothing

Two separate bugs:

  • ud-bazel-cache targeted ~/.cache/bazel, one directory inside the cache root, so its four siblings were container-local.
  • ud-go-cache targeted ~/go, but features/go bakes GOPATH=/go into the image ENV. Running the image with no volumes at all is the check that settles it:
$ docker run --rm --entrypoint /bin/bash vsc-busy-nash-… -lc 'go env GOPATH GOMODCACHE; ls /go; ls /home/vscode/go; ls /home/vscode/.cache'
/go
/go/pkg/mod
bin                                                   # /go holds only bin/
ls: cannot access '/home/vscode/go': No such file or directory
ls: cannot access '/home/vscode/.cache': No such file or directory

So ~/go is a path nothing in the container writes, while the real Go state at /go/pkg was discarded on every rebuild.

Neither fails loudly. The container just rebuilds those caches every time, which reads as "devcontainers are slow".

The fix

Two volumes, each at the root of what it persists:

  • ud-cache~/.cache
  • ud-go-pkg-cache/go/pkg

Why the whole of ~/.cache, not per-tool volumes. Per-tool only persists what someone remembered to enumerate, which is precisely the failure being fixed — bazelisk was silently missing and nobody noticed. Mounting the root means a tool that starts caching under XDG is covered with no edit. Nothing under it wants to stay ephemeral either: every entry is content-addressed or key-validated by its own tool, and CI builds cold, so a stale local cache can't reach main.

Why $GOPATH/pkg and not $GOPATH. /go/bin is image content — features/go builds ten tools there at image-build time — and Docker only seeds a volume from the image while the volume is empty. A volume over all of /go would freeze bin/ at whatever the image held on first mount, so a later feature bump would install tools nobody ever sees. Persisting it buys nothing anyway: post-create.sh reinstalls its six pinned tools over the image's copies on every create. pkg/ is the derived half of GOPATH, bin/ the artifact half.

Why not $GOMODCACHE alone. The checksum-db cache is a sibling of the module cache at /go/pkg/sumdb, and Docker creates the /go/pkg mountpoint parent root-owned, so the first go install dies on open /go/pkg/sumdb/sum.golang.org/latest: no such file or directory.

Nesting a ~/.cache volume under the existing ~/.cache/bazel one would have worked mechanically, but leaves two volumes with overlapping meaning and no axis separating them — bazel clean --expunge already gives per-tool discard from inside the container. So ud-bazel-cache is retired rather than kept.

post-create.sh's chown follows the new targets and is now guarded on current ownership. Unconditional chown -R over a warm ~/.cache is minutes spent re-asserting ownership that is already correct, and it would flatten the go feature's vscode:golang group.

Validation

All run in the devcontainer.

Persistence. Sentinels written in one container, then devcontainer up --remove-existing-container, then read back in the next:

--- container id (previous was fe3c0a7f11b8) ---
d6d13ad9db25
--- sentinels written by the PREVIOUS container ---
/home/vscode/.cache/SENTINEL-busy-nash   busy-nash-fd3c66 2026-08-19T18:46:31Z
/go/pkg/SENTINEL-busy-nash               busy-nash-fd3c66 2026-08-19T18:46:31Z
--- what survived, by size ---
1.9G  ~/.cache/go-build     61M  ~/.cache/bazelisk    13M  ~/.cache/pre-commit
544M  /go/pkg/mod           12K  /go/pkg/sumdb

/go/bin is not shadowed — the four tools post-create.sh does not install keep the image's date next to the six it does:

-rwxrwxr-x golint         2026-08-18 15:48    # image layer
-rwxrwxr-x goplay         2026-08-18 15:48    # image layer
-rwxrwxr-x revive         2026-08-18 15:48    # image layer
-rwxrwxr-x staticcheck    2026-08-18 15:48    # image layer
-rwxr-xr-x golangci-lint  2026-08-19 11:46    # post-create, pinned
-rwxr-xr-x gopls          2026-08-19 11:46    # post-create, pinned
… (ten total)

Ownership — the go feature's group survives on /go, and only the mount targets are re-owned:

/go vscode:golang 2775    /go/bin vscode:golang 2775
/go/pkg vscode:vscode 755    /home/vscode/.cache vscode:vscode 755

The chown is load-bearing for both targets, probed with fresh volumes against the bare image so no hook had run:

/go/pkg root:root 755
/home/vscode/.cache root:root 755
touch: cannot touch '/go/pkg/probe': Permission denied
touch: cannot touch '/home/vscode/.cache/probe': Permission denied

That corrected a claim carried over from the earlier attempt at this change, which said /go/pkg was the target that "actually arrives root-owned". The image has a directory at neither, so both do; the comment and the CLAUDE.md bullet say that now.

Warm effect. In the freshly recreated container: go build ./... 0.70s, pre-commit install-hooks 0.35s. The recreate's postCreateCommand did not re-download bazel 9.2.0 and uv sync reported Checked 6 packages rather than installing them — both were a download and an install on the pre-change container in the same session.

Suite. bazel test //... — 22/22 pass. pre-commit run --all-files — all seven hooks pass.

New tests are mutation-checked — each of these makes //.devcontainer:test_devcontainer_config fail, and reverting makes it pass:

  1. widening the go mount to /go (with the chown loop widened to match, so it is the root assertion catching it, not the agreement one)
  2. narrowing the cache mount back to ~/.cache/bazel
  3. dropping /go/pkg from the chown loop
  4. replacing the loop with a plain chown line

Notes for the reviewer

🤖 Generated with Claude Code

Both named volumes missed most of what they were meant to persist, and
neither failed loudly. Measured in a container built from current main:

  ~/.cache/bazel     mounted (ud-bazel-cache)
  ~/.cache/go-build  container-local, rebuilt every recreate
  ~/.cache/bazelisk  container-local -- a fresh 61M bazel download each time
  ~/.cache/pre-commit, ~/.cache/uv  container-local
  /go/pkg (mod 544M + sumdb)        container-local
  ~/go               mounted (ud-go-cache), written by nothing

Two separate bugs. `ud-bazel-cache` targeted one directory *inside* the
cache root, leaving its four siblings ephemeral. `ud-go-cache` targeted
`~/go`, but `features/go` bakes GOPATH=/go into the image ENV; running
the image with no volumes at all shows no `~/go` and no `/go/pkg`, so
that mount was writing nowhere while the real Go state was discarded.

Two volumes now, each at the root of what it persists: `ud-cache` ->
`~/.cache` and `ud-go-pkg-cache` -> `/go/pkg`.

Roots rather than per-tool directories, because per-tool only persists
what someone remembered to enumerate -- which is exactly what drifted
here, with `bazelisk` missing and nothing noticing.

Not all of `/go`: `bin/` there is image content (the feature builds ten
tools at image-build time) and Docker seeds a volume from the image only
while the volume is empty, so a volume over `/go` would freeze `bin/` at
first-mount contents and a later feature bump would install tools nobody
sees. Not `/go/pkg/mod` either: the checksum-db cache is its sibling at
`/go/pkg/sumdb`, and the root-owned mountpoint parent breaks `go install`
with `open /go/pkg/sumdb/...: no such file or directory`. `$GOPATH/pkg`
is the boundary that holds -- `pkg/` derived, `bin/` artifact.

post-create.sh's chown follows the new targets and is guarded on current
ownership: unconditional `chown -R` over a warm `~/.cache` is minutes of
re-asserting what is already correct, and on `/go` it would flatten the
go feature's `vscode:golang` group.

Verified in the devcontainer. Sentinels written in one container survived
`devcontainer up --remove-existing-container` in both volumes, alongside
1.9G go-build, 61M bazelisk, 13M pre-commit, 544M mod and 12K sumdb.
`/go/bin` is unshadowed -- golint/goplay/revive/staticcheck keep the
image's Aug 18 date next to the six post-create installs' Aug 19 -- and
`/go` keeps `vscode:golang 2775`. A probe with fresh volumes on the bare
image confirms both targets arrive root-owned and unwritable by `vscode`,
so the chown is load-bearing for each. `bazel test //...` is 22/22 and
`pre-commit run --all-files` passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread .devcontainer/test_devcontainer_config.py Outdated
Comment thread .devcontainer/post-create.sh Outdated
Comment thread .claude/CLAUDE.md Outdated
Review feedback on the three points raised.

The chown test asserted only what the loop iterates over. `chown_targets`
matched the `for volume_target in …; do` header alone, so a body that
chowned a literal -- or stopped chowning entirely -- kept returning the
right paths and kept the test green, while the container came up with
/go/pkg root-owned and the first `go install` died on permission denied.
The docstring had the rationale backwards too: reading only the header is
what lets the header and the body disagree.

It now matches the loop as a unit, header through `done`, and requires the
body to chown `"$volume_target"` on one line -- a `chown` somewhere and the
variable mentioned somewhere later is not a chown of it. Both bodies the
review named now raise, and the guarded real body still parses, which is
its own test since the chown is neither the first nor the only line.

The ownership guard tested `%u` while the chown sets `$(id -u):$(id -g)`,
so a GID-only drift was skipped. Now `%u:%g`, which costs nothing:
`sudo chgrp root /go/pkg` in a live container makes the old test skip and
the new one fire.

The guard reads the mount root, so it cannot heal a root-owned entry left
*inside* an otherwise-correct tree -- a tool run under sudo in here, or a
`chown -R` interrupted partway, which visits pre-order and so fixes the
root first. Keeping the guard, since a full walk costs roughly what it
saves, but the comment claimed a self-heal it only partly delivers; it now
says which case it heals, which it does not, and how to recover by hand.
The narrower statement made the earlier self-heal clause redundant, so
that paragraph is merged rather than left to drift against it.

CLAUDE.md said go-build was 967M, a figure inherited from the superseded
attempt; this PR measured 1.9G in both the gap table and the persistence
output, and the doc is the copy that outlives the PR.

Verified: both review mutations fail the test and the real script passes;
recreated the container, sentinels and all five caches survived, /go/bin
still unshadowed and /go still vscode:golang 2775; `bazel test //...`
22/22; `pre-commit run --all-files` passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread .devcontainer/test_devcontainer_config.py Outdated
Review follow-up. `_CHOWN_OF_LOOP_VAR_RE` ran against the raw loop body,
so a chown that had been *commented out* still satisfied it. Reproduced
against the committed code before fixing:

  PASSES  real body                     PASSES  commented-out chown
  PASSES  chown other path, same line   caught  chown literal
                                        caught  no chown at all

The realistic trigger is the one post-create.sh now sends people into: a
root-owned entry inside ~/.cache, someone comments the chown out to test
a hypothesis by hand, and does not restore it. `chown_targets` returns
both paths, the coupling test passes, and the next fresh create comes up
with both volumes root-owned -- the failure the body assertion exists to
catch.

Comment lines are stripped before the match now, per the suggestion.

Also closed the second row, which the review flagged as contrived and
worth leaving: `[^;&|\n]` bounds the gap between `chown` and the path, so
the path has to be an argument of the chown rather than a mention on a
line it happens to share. It is one character class, and without it the
assertion does not mean what its name says.

The `\n` in that class is load-bearing and cost a red test to find. A
negated character class matches newlines even without DOTALL, so the
first cut -- `[^;&|]` -- silently un-did the line-bound property the
comment above it is careful about, and
test_chown_targets_requires_the_chown_and_the_path_on_one_line caught it.
The comment now says why the `\n` is not redundant.

Four tests: the commented-out body, a chained command per separator, and
-- guarding the other direction -- a chown whose path is followed by
`|| true`, which is still a chown of that path and must keep passing.

Verified: every row of the review matrix now resolves as intended, the
real post-create.sh still parses to both targets, `bazel test //...`
22/22, `pre-commit run --all-files` passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Syndic
Syndic merged commit 164767a into main Aug 19, 2026
44 checks passed
@Syndic
Syndic deleted the claude/devcontainer-cache-roots branch August 19, 2026 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant