fix(container-run.sh): pin the dev/build container image by digest

.claude/CLAUDE.md

@@ -2,13 +2,20 @@
 
 All commands should be run from the repository root (`/ic`).
 
-Never manually edit `ci/container/TAG`. It is bumped only by the
+Never manually edit `ci/container/TAG`, `ci/container/ic-dev.digest` or
+`ci/container/ic-build.digest`. They are written only by the
 `container-autobuild.yml` GitHub Actions workflow, which builds the new
-dev-container image and pushes it to the registry *before* the tag change
-takes effect. Hand-editing it points the tag at an image that was never built
-or published, breaking CI for everyone. If the dev container image genuinely
+dev-container images and pushes them to the registry *before* committing the
+new tag and digests. `ci/container/container-run.sh` pulls the image by the
+committed digest: a malformed pin is refused outright, and a digest the
+registry does not serve cannot be pulled, so a hand-edited pin breaks the dev
+container for everyone. If the dev container image genuinely
 needs to change (e.g. `ci/container/Dockerfile` was edited), let that workflow
-bump `TAG` — don't do it in your commit.
+bump them — don't do it in your commit. Until its bot commit lands (this
+includes the sanctioned trick of putting a random string in `TAG` to force a
+rebuild), `TAG` does not match the working tree and `container-run.sh` refuses
+to run; `CONTAINER_RUN_ALLOW_UNPINNED=1` makes it build the image locally from
+the checkout instead.
 
 # Rust
 

.claude/skills/run-in-dev-container/SKILL.md

@@ -47,7 +47,13 @@ runtime's daemon isn't reachable it prints which command it tried.
 - The repo is mounted at `/ic` and that's the working directory, so invoke
   repo-local scripts with a relative path, e.g.
   `CONTAINER_RUNTIME=docker ./ci/container/container-run.sh ./path/to/script.sh`.
-- The image is pulled from `ghcr.io` on first use (large, one-time).
+- The image is pulled from `ghcr.io` on first use (large, one-time) by the
+  digest committed in `ci/container/ic-dev.digest` and verified against it
+  before it runs; registry tags are never pulled. If `ci/container/Dockerfile`,
+  `init.sh` or `files/*` differ from what `ci/container/TAG` was built from
+  (you edited them, or the autobuild's commit hasn't landed on your branch yet)
+  the script refuses to run; `CONTAINER_RUN_ALLOW_UNPINNED=1` makes it build the
+  image locally instead, which is slow.
 - Anything the command writes under `/ic` (or `~/.cache`) persists on the host,
   since those are bind-mounted.
 - Don't nest: the script refuses to run inside an existing container.

.github/workflows/ci-kickoff-manual.yml

@@ -95,6 +95,7 @@ jobs:
     with:
       commit-sha: ${{ inputs.commit-sha }}
       release-build: 'false'
+      pull-request: true
     permissions:
       actions: write
       contents: read

.github/workflows/ci-main.yml

@@ -12,6 +12,11 @@ on:
         required: false
         default: ''
         type: string
+      pull-request:
+        description: Whether this run is CI for a pull request even though the event is not pull_request (ci-kickoff-manual.yml dispatches CI for reviewed PRs, including fork PRs).
+        required: false
+        default: false
+        type: boolean
     outputs:
       release-build:
         description: Whether this run was a release build, i.e. uploaded artifacts to the CDN.
@@ -61,6 +66,8 @@ jobs:
             fuzzers:
               - ".github/workflows/ci-main.yml"
               - "ci/container/TAG"
+              - "ci/container/ic-build.digest"
+              - "ci/container/ic-dev.digest"
               - "bin/fuzzing/*.sh"
               - "**/BUILD.bazel"
               - "**/*.bzl"
@@ -822,6 +829,8 @@ jobs:
               - "**/*.lock"
               - "ci/scripts/rust-lint.sh"
               - "ci/container/TAG"
+              - "ci/container/ic-build.digest"
+              - "ci/container/ic-dev.digest"
               - ".github/**"
       - name: Run Cargo ${{ matrix.name }} Linux
         if: |
@@ -872,16 +881,59 @@ jobs:
     name: Test Container Run ${{ matrix.name }}
     runs-on: ubuntu-latest
     timeout-minutes: 30
+    # The docker leg is informational until it has been seen green.
+    continue-on-error: ${{ matrix.experimental == true }}
     strategy:
+      fail-fast: false
       matrix:
         include:
-          - name: TestCase 1
+          # Happy paths: the pinned image is pulled by digest, verified and run.
+          - name: ic-dev default
+            setup: ""
             command: ""
-          - name: TestCase 2
+            expect: success
+            runtime: podman
+          - name: ic-build
+            setup: ""
             command: "--image ic-build bash"
+            expect: success
+            runtime: podman
+          # A local image squatting on the tag must be ignored: the pinned digest
+          # is pulled and run instead, and the tag is re-pointed to it (checked by
+          # the "Verify the poisoned tag ..." step below). The squatter is an
+          # empty image that cannot run anything, so a successful `true` already
+          # proves the pinned image ran.
+          - name: poisoned local tag
+            setup: |
+              tar -cf empty.tar -T /dev/null
+              repo=ghcr.io/dfinity
+              sudo podman import empty.tar "$repo/ic-dev:$(cat ci/container/TAG)"
+            command: "true"
+            expect: success
+            runtime: podman
+          # A tampered pin must be refused before anything is pulled or built.
+          - name: malformed pin refused
+            setup: echo not-a-digest > ci/container/ic-dev.digest
+            command: "true"
+            expect: "refusing to pull an unpinned image"
+            runtime: podman
+          # A well-formed pin the registry does not serve: the pull fails and, by
+          # default, neither a local build nor a cached image is used.
+          - name: unknown digest refused
+            setup: printf 'sha256:%064d\n' 0 > ci/container/ic-dev.digest
+            command: "true"
+            expect: "refusing to build an unpinned image locally"
+            runtime: podman
+          - name: docker runtime
+            setup: ""
+            command: "true"
+            expect: success
+            runtime: docker
+            experimental: true
     steps:
       - *checkout
       - name: Workaround for actions/runner-images#14473 (crun pinning)
+        if: matrix.runtime == 'podman'
         run: |
           # Workaround for actions/runner-images#14473:
           # Podman 5.x on ubuntu-24.04/22.04 picks /usr/bin/crun (1.14.1) over
@@ -904,14 +956,128 @@ jobs:
           ref: ${{ inputs.commit-sha }}
           filters: |
             container-run:
+              - '.github/workflows/ci-main.yml'
               - '.github/workflows/ci-pr-only.yml'
               - '.github/workflows/container-autobuild.yml'
               - 'ci/container/**'
-      - name: Test Container Run
+            pins:
+              - '.devcontainer/**'
+              - '.github/workflows/**'
+      - name: Decide whether to run
+        id: decide
         if: |
           steps.filter.outputs.container-run == 'true' ||
           env.BRANCH_NAME == 'master' ||
           startsWith(env.BRANCH_NAME, 'rc--') ||
           startsWith(env.BRANCH_NAME, 'hotfix-')
+        run: echo "run=true" >> "$GITHUB_OUTPUT"
+      - name: Check the digest pins are consistent
+        if: steps.decide.outputs.run == 'true' || steps.filter.outputs.pins == 'true'
         run: |
-          ./ci/container/container-run.sh ${{ matrix.command }}
+          # ci/container/*.digest, the ic-dev pin in .devcontainer/ and the ic-build
+          # pins in the workflows are all written by container-autobuild.yml and
+          # must agree. Never compare against what the registry serves under
+          # ci/container/TAG: that would make the mutable tag the oracle again.
+          #
+          # container-autobuild.yml rewrites every "<prefix>ic-build[:@]..." string
+          # under .github/workflows/ with sed, so the image names below are kept
+          # in a variable to keep this script from being rewritten too.
+          set -euo pipefail
+          prefix="ghcr.io/dfinity/"
+          dev_digest="$(cat ci/container/ic-dev.digest)"
+          build_digest="$(cat ci/container/ic-build.digest)"
+          for digest in "$dev_digest" "$build_digest"; do
+            if ! [[ $digest =~ ^sha256:[0-9a-f]{64}$ ]]; then
+              echo "::error::malformed digest '$digest' in ci/container/*.digest (they are written by container-autobuild.yml)"
+              exit 1
+            fi
+          done
+          # Collect every reference (":tag" or "@digest", well-formed or not) so
+          # that a reference changed to a mutable tag or a malformed digest is
+          # rejected too, not only a mismatching well-formed one.
+          dev_refs="$(grep -oE "${prefix}ic-dev[:@][^\"'[:space:]]+" .devcontainer/devcontainer.json | sort -u)"
+          if [ "$dev_refs" != "${prefix}ic-dev@$dev_digest" ]; then
+            echo "::error file=ci/container/ic-dev.digest::every ic-dev reference in .devcontainer/devcontainer.json must be ${prefix}ic-dev@$dev_digest (ci/container/ic-dev.digest; both are written by container-autobuild.yml), found:"
+            echo "${dev_refs:-<none>}"
+            exit 1
+          fi
+          build_refs="$(grep -rhoE "${prefix}ic-build[:@][^\"'[:space:]]+" .github/workflows/ | sort -u)"
+          if [ "$build_refs" != "${prefix}ic-build@$build_digest" ]; then
+            echo "::error file=ci/container/ic-build.digest::every ic-build reference under .github/workflows/ must be ${prefix}ic-build@$build_digest (ci/container/ic-build.digest; all are written by container-autobuild.yml), found:"
+            echo "${build_refs:-<none>}"
+            exit 1
+          fi
+      - name: Check ci/container/TAG matches the container inputs
+        id: sync
+        if: steps.decide.outputs.run == 'true'
+        env:
+          # pull_request events, and CI dispatched for a PR by ci-kickoff-manual.yml
+          CI_PULL_REQUEST: ${{ github.event_name == 'pull_request' || inputs.pull-request }}
+        run: |
+          # When TAG is out of sync there is no reviewed digest for the working
+          # tree and container-run.sh refuses to run (it only builds locally with
+          # CONTAINER_RUN_ALLOW_UNPINNED=1), so every leg below would fail. On a
+          # pull request that state is transient:
+          # the container-autobuild.yml bot commit brings TAG and *.digest in sync
+          # and re-triggers CI (on fork PRs it cannot, and they build locally).
+          # Anywhere else (push to master, the merge queue, release-testing on
+          # rc--/hotfix-, a manual run for a branch) it means that commit never
+          # landed, so fail.
+          set -euo pipefail
+          computed="$(ci/container/get-image-tag.sh)"
+          pinned="$(cat ci/container/TAG)"
+          if [ "$computed" = "$pinned" ]; then
+            echo "in_sync=true" >> "$GITHUB_OUTPUT"
+          elif [ "$CI_PULL_REQUEST" = true ]; then
+            echo "::notice file=ci/container/TAG::ci/container/TAG ($pinned) does not match the container inputs (computed $computed); skipping the container-run tests: the Container IC Build Image workflow will commit TAG and *.digest to this branch and re-trigger CI"
+            echo "in_sync=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "::error file=ci/container/TAG::ci/container/TAG ($pinned) does not match the container inputs (computed $computed): the container-autobuild.yml commit did not land on $BRANCH_NAME ($CI_EVENT_NAME)"
+            exit 1
+          fi
+      - name: Set up "${{ matrix.name }}"
+        if: steps.sync.outputs.in_sync == 'true' && matrix.setup != ''
+        run: ${{ matrix.setup }}
+      - name: Test Container Run
+        if: steps.sync.outputs.in_sync == 'true'
+        env:
+          CONTAINER_RUNTIME: ${{ matrix.runtime }}
+        run: |
+          set -euo pipefail
+          if [ "${{ matrix.expect }}" = success ]; then
+            ./ci/container/container-run.sh ${{ matrix.command }}
+            exit 0
+          fi
+          # A tampered pin must fail fast: not succeed, not hit the timeout, and
+          # not fall through to a local build that merely times the job out.
+          set +e
+          out="$(timeout 300 ./ci/container/container-run.sh ${{ matrix.command }} 2>&1)"
+          rc=$?
+          set -e
+          printf '%s\n' "$out"
+          if [ "$rc" -eq 0 ] || [ "$rc" -eq 124 ]; then
+            echo "::error::expected container-run.sh to refuse quickly, got exit code $rc"
+            exit 1
+          fi
+          if ! grep -qF -- "${{ matrix.expect }}" <<<"$out"; then
+            echo "::error::expected the output to contain '${{ matrix.expect }}'"
+            exit 1
+          fi
+          repo=ghcr.io/dfinity
+          if [ -n "$(sudo podman images -q "$repo/ic-dev")" ]; then
+            echo "::error::an ic-dev image was pulled or built despite the refusal"
+            exit 1
+          fi
+      - name: Verify the poisoned tag was re-pointed to the verified image
+        if: steps.sync.outputs.in_sync == 'true' && matrix.name == 'poisoned local tag'
+        run: |
+          set -euo pipefail
+          repo=ghcr.io/dfinity
+          tag="$repo/ic-dev:$(cat ci/container/TAG)"
+          want="$repo/ic-dev@$(cat ci/container/ic-dev.digest)"
+          digests="$(sudo podman image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$tag")"
+          printf '%s\n' "$digests"
+          if ! grep -qxF "$want" <<<"$digests"; then
+            echo "::error::$tag does not carry the pinned digest $want after the run"
+            exit 1
+          fi

.github/workflows/container-autobuild.yml

@@ -45,7 +45,8 @@ jobs:
           if grep $image_tag ci/container/TAG; then
             echo "build_image=false" >> "$GITHUB_OUTPUT"
             echo "Update with this image already in this PR!"
-            echo "Modify ci/container/TAG with random string to trigger new build."
+            echo "Modify ci/container/TAG with a random string to trigger a new build; until this"
+            echo "workflow's commit lands, container-run.sh on that commit builds the image locally."
           else
             echo "build_image=true" >> "$GITHUB_OUTPUT"
             echo "image_tag=$image_tag" >> "$GITHUB_OUTPUT"
@@ -160,10 +161,25 @@ jobs:
           set -xeuo pipefail
 
           IMAGE_PREFIX="ghcr.io/dfinity/"
-          IMG_NAME_IC_BUILD="${IMAGE_PREFIX}ic-build@${{ needs.ic-build-image.outputs.ic-build-imagedigest }}"
-          IMG_NAME_IC_DEV="${IMAGE_PREFIX}ic-dev@${{ needs.ic-build-image.outputs.ic-dev-imagedigest }}"
-
+          IC_BUILD_DIGEST='${{ needs.ic-build-image.outputs.ic-build-imagedigest }}'
+          IC_DEV_DIGEST='${{ needs.ic-build-image.outputs.ic-dev-imagedigest }}'
+          # container-run.sh pulls by these digests and refuses a malformed one, so
+          # fail here rather than committing a pin that breaks every container run.
+          for digest in "$IC_BUILD_DIGEST" "$IC_DEV_DIGEST"; do
+            if ! [[ $digest =~ ^sha256:[0-9a-f]{64}$ ]]; then
+              echo "::error::malformed image digest '$digest'"
+              exit 1
+            fi
+          done
+          IMG_NAME_IC_BUILD="${IMAGE_PREFIX}ic-build@${IC_BUILD_DIGEST}"
+          IMG_NAME_IC_DEV="${IMAGE_PREFIX}ic-dev@${IC_DEV_DIGEST}"
+
+          # TAG and the *.digest files are read by ci/container/container-run.sh,
+          # which pulls the images by digest; they live directly under
+          # ci/container/ (never in ci/container/files/, an input of get-image-tag.sh).
           echo '${{ needs.build-image-prep.outputs.image_tag }}' > ci/container/TAG
+          echo "$IC_BUILD_DIGEST" > ci/container/ic-build.digest
+          echo "$IC_DEV_DIGEST" > ci/container/ic-dev.digest
           pushd .devcontainer
           sed -i -E "s|${IMAGE_PREFIX}ic-dev(:\|@)[^\"]{5,}|$IMG_NAME_IC_DEV|g" -- *
           popd

ci/container/README.md

@@ -84,9 +84,18 @@ sudo podman run --pids-limit=-1 -it --rm --privileged --network=host --cgroupns=
   --mount type=bind,source=/home/john/.local/share/fish,target=/home/ubuntu/.local/share/fish \
   --mount type=bind,source=/home/john/.zsh_history,target=/home/ubuntu/.zsh_history \
   -v /tmp/ssh-XXXXQAO7kF/agent.113731:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent -w /ic \
-  ghcr.io/dfinity/ic-dev:221b79c4f4a966eae67a3f9ef7f20f4c5583d5bc38df17c94128804687a84c29 /usr/bin/fish
+  --pull=never \
+  ghcr.io/dfinity/ic-dev@sha256:3e028a468490dddda2b255da986c8ea2fe9c2dd2a143d4557c9bd5bb68c8218a /usr/bin/fish
 ```
 
+### Image pinning
+
+`container-run.sh` never pulls an image by registry tag: the tags on `ghcr.io/dfinity/ic-dev` and `ghcr.io/dfinity/ic-build` are mutable and can be re-pointed without review. Instead it pulls `ghcr.io/dfinity/<image>@sha256:...` using the digest committed in [ic-dev.digest](ic-dev.digest) / [ic-build.digest](ic-build.digest), checks that the local image carries that digest, and starts it with `--pull=never`. Like [TAG](TAG), the `.digest` files are written only by the `container-autobuild.yml` workflow. Never edit them by hand: a malformed pin makes the script refuse to run, and a digest the registry does not serve cannot be pulled, so the script refuses as well.
+
+`TAG` is the hash of `Dockerfile`, `init.sh` and `files/*`. If your working tree hashes to something else (you edited one of those files, or the autobuild's bot commit has not landed on your branch yet) no reviewed digest exists for it and the script refuses to run. To build the image locally from your checkout instead (slow, and it needs network), set `CONTAINER_RUN_ALLOW_UNPINNED=1`; an existing local `<image>:<tag>` is then reused. Once the bot commit lands, `git pull` and re-run to get the registry image. The autobuild cannot publish for pull requests from forks, so contributors working on a fork need `CONTAINER_RUN_ALLOW_UNPINNED=1` after touching those files.
+
+By default the script never runs an image that is not verified against the pin: no local build, and no reuse of a cached image when the pinned pull fails (e.g. offline). `CONTAINER_RUN_ALLOW_UNPINNED=1` opts out of that for one run and is announced with a warning; leave it unset when verifying release artifacts.
+
 ### How to use custom config
 
 User can create config `$HOME/.container-run.conf`, with `podman run` arguments, that provide way of adding custom bind-mounts etc. Config file requires array variable `PODMAN_RUN_USR_ARGS` with arguments accepted by `podman run` (see `podman run --help`). See example config from `.container-run.conf` below: