fix(ci): give each checkout its own bazel output base in container-run.sh

ci/container/README.md

@@ -99,9 +99,22 @@ PODMAN_RUN_USR_ARGS=(
 )
 ```
 
-### How to run parallel bazel tests
+### Running containers from several checkouts (git worktrees, clones) at once
 
-By default `container-run.sh` bind-mounts `~/.cache` which is used for (output_base)[https://bazel.build/docs/user-manual#output-base]. If you need to run 2nd build/test in parallel but not interfere with the 1st one, follow the steps below.
+Bazel derives its default output base from the workspace path, and every checkout is mounted at `/ic` with the same `~/.cache`, so containers started from different checkouts would share one output base. Bazel cannot recognize a server that runs in another container (each container has its own PID namespace), so the second container would start another server in the same output base and the first one would die with `Server terminated abruptly (error code: 14, ...)`.
+
+`container-run.sh` therefore gives every checkout its own output base, `~/.cache/bazel/_bazel_ubuntu/<basename>-<hash of the host path>`, by pointing `BAZELRC` at a generated rc file under `~/.cache/container-run/`. The install base, the repository cache and the repo contents cache stay shared, and the rest of the repository's bazel configuration (`.bazelrc`, `user.bazelrc`) still applies. Containers started from different checkouts can run bazel concurrently.
+
+Notes:
+
+- The first run from a checkout builds from scratch (the remote cache helps). The previously shared output base `~/.cache/bazel/_bazel_ubuntu/6d065581cce7ad9076e3b8db2b3afaf0` can be deleted to reclaim disk space once no container uses it anymore.
+- Two containers started from the *same* checkout still share an output base and must not run bazel at the same time. For a second shell in a running container use `sudo podman exec -it <container> bash` (on a devenv: `sudo podman --root /hoststorage/podman-root exec -it <container> bash`).
+- The injected rc file is read after the workspace `.bazelrc` and `user.bazelrc`, so a `startup --output_base` in `user.bazelrc` is overridden inside the container.
+- `~/.cache/cargo` (`CARGO_TARGET_DIR`) is still shared between all checkouts; concurrent cargo builds serialize on its lock and invalidate each other's artifacts.
+
+Linked git worktrees (`git worktree add`) are supported: the main repository's `.git` directory is bind-mounted at its host path so that git works inside the container. There `git worktree list` shows the linked worktrees as `prunable` because their checkouts are not visible in the container, so never run `git worktree prune`, `repair`, `move` or `remove` inside a container (gc's automatic worktree pruning is disabled via `gc.worktreePruneExpire=never`).
+
+To isolate everything, including the install base, repository cache and cargo target dir, use a separate cache directory instead:
 
 ```bash
 mkdir ~/.cache2

ci/container/container-run.sh

@@ -205,6 +205,58 @@ RUNTIME_RUN_ARGS=(
     --mount type=tmpfs,target="/tmp/containers" # expected by ic-os build
 )
 
+# Give every checkout its own bazel output base.
+#
+# Bazel derives the default output base from the md5 of the workspace path.
+# Every checkout is mounted at the same path ($WORKDIR) with the same cache
+# dir, so containers started from different checkouts (git worktrees or
+# clones) would all use one output base. Each container also has its own PID
+# namespace, so bazel's client cannot recognize the other container's server
+# (it verifies the server pid via /proc), starts a second server in the same
+# output base, and the first server then kills itself: its client fails with
+# "Server terminated abruptly (error code: 14 ...)". Keying the output base on
+# the host path of the checkout avoids this while keeping the install base,
+# the repository cache and the repo contents cache (all of which live in the
+# shared output_user_root) shared between checkouts.
+#
+# Bazel reads the rc files named in $BAZELRC in addition to the workspace
+# .bazelrc (including its user.bazelrc import) and ~/.bazelrc, so the rest of
+# the repository's bazel configuration still applies. The variable only
+# exists inside the container, so host-side bazel invocations are unaffected.
+# Keep only characters that are safe in the rc line, in $BAZELRC (comma-separated) and in
+# a filename; the path hash below keeps the key unique.
+REPO_NAME="$(printf '%s' "$(basename "$REPO_ROOT")" | LC_ALL=C tr -c 'A-Za-z0-9._-' '_' | cut -c1-64)"
+OUTPUT_BASE_KEY="$REPO_NAME-$(printf '%s' "$REPO_ROOT" | sha256sum | cut -c1-8)"
+OUTPUT_BASE="$CTR_CACHE_DIR/bazel/_bazel_$CTR_USER/$OUTPUT_BASE_KEY"
+BAZELRC_REL="container-run/$OUTPUT_BASE_KEY.bazelrc" # relative to the cache dir
+mkdir -p "$(dirname "$CACHE_DIR/$BAZELRC_REL")"
+echo "startup --output_base=$OUTPUT_BASE" >"$CACHE_DIR/$BAZELRC_REL"
+RUNTIME_RUN_ARGS+=(-e BAZELRC="$CTR_CACHE_DIR/$BAZELRC_REL")
+eprintln "Using bazel output base '$OUTPUT_BASE'"
+
+# Support linked git worktrees (`git worktree add`).
+#
+# A linked worktree's .git is a file pointing into the main repository's .git
+# directory on the host, which is not otherwise visible in the container, so
+# git (and everything that uses it: --config=stamped, rust-lint.sh, ic-admin's
+# build script under cargo, ...) would fail with "not a git repository".
+# Bind-mount the common git dir at its host path so that the pointer resolves.
+#
+# Inside such a container `git worktree list` reports the linked worktrees as
+# "prunable" because their host checkout paths are not visible, so never run
+# `git worktree prune|repair|move|remove` in the container. For the same
+# reason gc's automatic worktree pruning is disabled via GIT_CONFIG_*.
+GIT_COMMON_DIR="$(cd "$REPO_ROOT" && realpath "$(git rev-parse --git-common-dir)")"
+if [ "$GIT_COMMON_DIR" != "$REPO_ROOT/.git" ]; then
+    eprintln "Detected linked git worktree; mounting '$GIT_COMMON_DIR'"
+    RUNTIME_RUN_ARGS+=(
+        --mount type=bind,source="$GIT_COMMON_DIR",target="$GIT_COMMON_DIR"
+        -e GIT_CONFIG_COUNT=1
+        -e GIT_CONFIG_KEY_0=gc.worktreePruneExpire
+        -e GIT_CONFIG_VALUE_0=never
+    )
+fi
+
 # Privilege/isolation flags required by the IC-OS guest build, per runtime.
 if [ "$RUNTIME" = docker ]; then
     # Under docker the IC-OS build runs (rootless) podman *inside* this