[brbrr]

User description

Summary

The security scan of nethermind/juno:v0.16.7 reports 158 CVE IDs (17 Critical, 51 High, 73 Medium, 17 Low). Almost all of them come from the Debian 12 base image and from packages that juno does not use (curl, gawk, grep, and the libstdc++ that libjemalloc2 pulls in). This PR removes them.

• Runtime image: the final stage is FROM scratch with a Wolfi root filesystem. A new base stage builds it with apko from the public Wolfi repository (packages.wolfi.dev/os, checked with the Wolfi signing key). The image has glibc, libgcc, CA certificates and tzdata, but no shell and no package manager.
  • docker/apko.yaml lists the packages. docker/apko.lock.json pins the exact version of each package for amd64 and arm64, so a build from a given commit is reproducible.
  • make apko-lock updates the lock file to the latest Wolfi packages.
  • The base stage runs on the build platform, so a cross-arch build needs no emulation for it.
• jemalloc: the builder removes libjemalloc.so, so -ljemalloc links libjemalloc.a. The binary now needs only libc, libm, libgcc_s and ld-linux. The narenas:2 config still applies.
• No apt layer: curl, gawk, grep, libjemalloc and libstdc++ are not in the image now.
• OCI labels: the image sets its own title, vendor, source and licenses labels.
• golang.org/x/net v0.58.0 → v0.60.0: fixes CVE-2026-78659, -78660, -78663 and -78669 in x/net/http2 (used through grpc). The Go 1.27.2 builder fixes the stdlib part of these CVEs.
• docker-compose.yml: the healthcheck runs juno --help, because the image has no sleep.
• User: juno still runs as root, as before. Data volumes from earlier images stay writable.

Image size: 248MB → 141MB (arm64), 150MB (amd64).

Breaking change

The image has no shell and no utilities. These no longer work:

• docker exec -it <container> sh. Use a debug container that shares the namespaces (docker run --pid container:juno --network container:juno busybox sh, or kubectl debug --target=juno), or docker cp.
• command: ["sh", "-c", "juno ..."], or any other script in the container. Run juno directly with a list of args. Kubernetes expands $(VAR) in args, and juno reads JUNO_* environment variables.
• Probes or healthchecks that run curl or other tools in the container. HTTP probes on /live and /ready are not affected.

The ArgoCD workloads that started juno through a shell were changed in NethermindEth/argocd#10983 (merged).

CVE result

Trivy 0.69.3, same DB, old and new image:

Image	OS	OS package IDs	juno binary IDs	IDs from the scan list
nethermind/juno:v0.16.7	Debian 12	165	14	155 of 158 found
this PR	Wolfi	0	1	0 of 158

The remaining Go finding is GO-2026-5932 (golang.org/x/crypto v0.57.0). It has no fix yet.

Trivy does not report 3 IDs from the scan list:

• CVE-2025-8941 (PAM): there is no PAM in the new image.
• CVE-2022-23328 (go-ethereum): the scanner matches all geth versions. This needs a scanner exception.
• CVE-2023-26248 (go-libp2p-kad-dht): a design weakness with no fix. This needs a scanner exception.

Testing

On arm64 with the image built from this branch, and on amd64 with the same Wolfi base and juno binary (CI builds the full amd64 image):

• The juno binary needs only libm, libgcc_s, libc and ld-linux, has Go 1.27.2 and x/net v0.60.0, and the jemalloc narenas:2 config applies.
• The image has no /bin/sh and no libstdc++.
• A Sepolia sync on a root-owned volume works, and starknet_call of ETH name() returns 0x4574686572 ("Ether").
• TZ=Europe/Berlin gives log timestamps in +02:00.
• Trivy detects Wolfi and reports 0 OS vulnerabilities.
• Two apko build runs from the same lock file give the same SHA-256.

In CI, smoke_test_100_blocks passes with the amd64 image. dev_deploy_test / deploy fails because the dev node does not get the new image. It fails the same way on all branches since 2026-10-13, so this PR does not cause it.

---

PR Type

Enhancement

---

Description

• Replace Debian runtime base with a shell-free Wolfi rootfs built via apko, dropping most scanned CVEs

• Pin exact Wolfi package versions in docker/apko.lock.json; add make apko-lock to refresh

• Link jemalloc statically in the builder; final image is built FROM scratch with OCI labels

• Bump golang.org/x/net to v0.60.0, fixing http2 CVEs reached through grpc

---

File Walkthrough

	Relevant files
Configuration changes	4 files Makefile Add apko-lock target for pinning Wolfi packages                    Makefile Adds APKO_VERSION variable (v1.4.11), kept in sync with the Dockerfile Adds apko-lock target that runs apko to regenerate docker/apko.lock.json with the latest Wolfi packages +6/-0      docker-compose.yml Fix healthcheck for shell-free runtime image                          docker-compose.yml Replaces the healthcheck command sleep 4 with juno --help, since the new runtime image contains no shell +1/-1      apko.lock.json Pin Wolfi package versions for reproducible image builds  docker/apko.lock.json New lock file pinning exact versions and checksums of each Wolfi package for x86_64 and aarch64 Embeds the Wolfi signing key and packages.wolfi.dev repository references for reproducible, verified builds +298/-0  apko.yaml Add apko config for Wolfi runtime filesystem                          docker/apko.yaml New apko manifest defining the runtime filesystem packages: wolfi-baselayout, ca-certificates-bundle, glibc-2.44, ld-linux-2.44, libgcc and tzdata Declares x86_64 and aarch64 target architectures +19/-0
Enhancement	1 files Dockerfile Build scratch-based runtime image from apko Wolfi rootfs  Dockerfile Builder stage removes shared libjemalloc.so so -ljemalloc links the static libjemalloc.a New base stage installs apko and builds a Wolfi rootfs from docker/apko.yaml and the lock file, extracting its single layer on the build platform (no emulation for cross-arch) Final stage switches from debian:bookworm-slim to scratch, copying only the apko rootfs and the juno binary Sets SSL_CERT_FILE and adds OCI title, vendor, source and licenses labels +41/-11
Dependencies	2 files go.mod Upgrade golang.org/x/net for http2 CVE fixes                          go.mod Bumps golang.org/x/net from v0.58.0 to v0.60.0, fixing CVEs in x/net/http2 used through grpc +1/-1      go.sum Update checksums for x/net dependency bump                              go.sum Updates dependency checksums to match the golang.org/x/net v0.60.0 upgrade and its transitive module changes

---

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.87%. Comparing base (77f38a6) to head (8c3be88).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4187      +/-   ##
==========================================
- Coverage   80.91%   80.87%   -0.04%     
==========================================
  Files         509      509              
  Lines       39161    39161              
==========================================
- Hits        31686    31671      -15     
- Misses       7469     7484      +15     
  Partials        6        6              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Claude finished @brbrr's task in 8m 7s —— View job

---

Review complete

• Gather context (PR diff, changed files)
• Review Dockerfile changes
• Review docker-compose.yml changes
• Review go.mod/go.sum changes
• Post findings

Summary

This is a well-scoped, well-documented infra change (Debian → Chainguard Wolfi runtime base, jemalloc static-link trick, x/net bump, healthcheck fix). No correctness bugs, concurrency issues, or protocol-path changes — it only touches Dockerfile, docker-compose.yml, go.mod/go.sum.

Nit

• Dockerfile:53 — USER root re-introduces root execution in the final image, partially offsetting the hardening this PR delivers. The PR body already flags this as a known trade-off with non-root as a follow-up; left an inline comment suggesting a tracking issue/comment link so that follow-up isn't lost. Not a blocker — the root-owned-volumes compatibility reason is reasonable for now.

Verified separately:

• jemalloc/jemalloc.go's pkg-config: jemalloc cgo link only needs the .a left after rm libjemalloc.so (Dockerfile:16); the dynamic Debian libjemalloc.so.2 is what was pulling in libstdc++, matching the PR's stated rationale.
• docker-compose.yml's healthcheck swap to juno --help is a safe no-op probe (cobra's --help path returns before touching DB/network config, confirmed in cmd/juno/juno.go).
• go.mod/go.sum x/net bump is indirect-only (pulled via grpc), version and hash are consistent between the two files.

No inline blockers. Nice work on the CVE reduction and documentation.

[github-actions]

Claude finished @brbrr's task in 18m 3s —— View job

---

Review complete

• Gather context (PR diff, changed files)
• Review Dockerfile changes (apko-based build)
• Review Makefile changes
• Review docker/apko.yaml and docker/apko.lock.json
• Review docker-compose.yml changes
• Review go.mod/go.sum changes
• Post findings

This round switches the final-stage approach from the previously-reviewed digest-pinned cgr.dev/chainguard/glibc-dynamic base to building the runtime filesystem from pinned Wolfi packages via apko, assembled into a FROM scratch final stage. The core idea (minimal rootfs + statically-linked jemalloc + binary) is sound and well documented (Dockerfile, docker/apko.yaml, Makefile all cross-reference APKO_VERSION to stay in sync).

Nit

• Dockerfile:62-65 — the single layer produced by apko build is extracted from manifest.json with a one-line sed regex rather than a JSON-aware tool. It works for apko's current compact, single-layer output, but if a future apko bump pretty-prints the manifest or emits more than one layer, the regex silently matches nothing and tar -xzf "" fails with a generic "cannot open" error instead of a clear message pointing at the real cause. Left an inline comment with a suggested jq-based alternative. Not a blocker given the pinned APKO_VERSION=v1.4.11 works as verified in the PR's own testing.
• The final stage now has no USER/accounts: directive at all (apko.yaml has no accounts: stanza), so the container still runs as root — same net effect as the previous round's explicit USER root, but the rationale (root-owned data volumes from earlier images) is no longer called out anywhere near the final stage. Worth a one-line comment near FROM scratch AS final so the reasoning isn't lost, though this doesn't change the security posture versus the prior round.

No correctness, concurrency, or protocol-path issues — this PR only touches build tooling (Dockerfile, Makefile, docker/apko.*, docker-compose.yml, go.mod/go.sum). The golang.org/x/net bump and docker-compose.yml healthcheck change are unchanged from the previously-reviewed version and remain fine.
· branch: chore/docker-image-hardening

[github-actions]

Claude finished @brbrr's task in 16m 58s —— View job

---

Review complete

• Gather context (PR diff, changed files)
• Review Dockerfile changes
• Review Makefile changes
• Review docker/apko.yaml and docker/apko.lock.json
• Review docker-compose.yml changes
• Review go.mod/go.sum changes
• Post findings

Summary

This PR only touches build tooling (Dockerfile, Makefile, docker/apko.*, docker-compose.yml, go.mod/go.sum) — no changes to consensus/, sync/, p2p/, RPC handlers, or other Go runtime code. No correctness bugs, concurrency issues, or protocol-path concerns apply here.

Latest commit (754d7fda4) replaces the earlier sed-based manifest parsing (flagged as a nit in the prior review round) with a python3-based JSON parser that also asserts len(images) == 1 and len(layers) == 1, failing loudly instead of silently if a future apko bump changes the manifest shape. That addresses the previously raised concern — no blocker remains there.

Verified in this pass:

• jemalloc/jemalloc.go:6 uses #cgo pkg-config: jemalloc, which resolves to -ljemalloc; removing the shared libjemalloc.so in the builder (Dockerfile:16) correctly forces the linker to use the static .a, consistent with the PR's stated rationale for dropping the runtime jemalloc/libstdc++ dependency.
• docker/apko.yaml and docker/apko.lock.json package names/versions/architectures are consistent (glibc-2.44, ld-linux-2.44, libgcc, ca-certificates-bundle, tzdata, wolfi-baselayout — pinned for both x86_64 and aarch64).
• Makefile:126 and Dockerfile:53 both pin APKO_VERSION=v1.4.11 with "keep in sync" comments in both places — a minor manual-sync nit, but already self-documented and low risk given it's a build-time-only pin.
• CI's "Container Security" job (docker build . + Trivy scan) passed on this commit, which exercises the full apko base-stage build end-to-end (including python3 availability in golang:1.27-bookworm and the --arch ${TARGETARCH} apko invocation) — confirming the build works as described.
• docker-compose.yml's healthcheck swap to juno --help and the golang.org/x/net v0.58.0 → v0.60.0 bump (indirect-only, via grpc) are unchanged from the previously-reviewed version and remain fine.

No inline comments this round — nothing new to flag beyond what was already addressed. Nice iteration on hardening the manifest parsing.
· branch: chore/docker-image-hardening

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

⚠️ Risk level: Medium

📂 Priority files Dockerfile docker/apko.yaml Makefile docker-compose.yml

🏅 Score: 70

🧪 No relevant tests

🔒 No security concerns identified

[github-actions]

Claude finished @brbrr's task in 4m 22s —— View job

---

Review complete

• Gather context (PR diff, changed files)
• Review Dockerfile changes (latest commit: install python3 explicitly)
• Review Makefile changes
• Review docker/apko.yaml and docker/apko.lock.json
• Review docker-compose.yml changes
• Review go.mod/go.sum changes
• Post findings

Summary

This PR only touches build tooling (Dockerfile, Makefile, docker/apko.*, docker-compose.yml, go.mod/go.sum) — no changes under consensus/, sync/, p2p/, RPC handlers, or other Go runtime code, so none of the correctness/concurrency/protocol concerns from the review focus apply.

Blocker from the prior round — resolved

• The last review flagged that Dockerfile's base stage ran python3 -c ... to parse manifest.json, but golang:1.27-bookworm doesn't ship python3 by default, which would fail the build. Commit d066578d2 (Dockerfile:56-59) now installs python3 explicitly via apt-get install -y --no-install-recommends python3 before that step runs. This resolves the blocker — no further action needed here.

Nit (carried over, non-blocking)

• Dockerfile:52 / Makefile:126 still pin APKO_VERSION=v1.4.11 in two places with "keep in sync" comments rather than a single source of truth (e.g. a build-arg passed from the Makefile, or an .env file read by both). Low risk since both are already cross-referenced, but worth consolidating if apko gets bumped often.
• The final scratch image still runs as root (no USER/accounts: in apko.yaml), same as previously discussed trade-off — the PR body already documents this as intentional (root-owned data volumes from earlier images), so not re-flagging as a new issue.

Verified

• docker/apko.yaml and docker/apko.lock.json package sets/versions/architectures are consistent (glibc-2.44, ld-linux-2.44, libgcc, ca-certificates-bundle, tzdata, wolfi-baselayout, pinned for both x86_64 and aarch64).
• jemalloc/jemalloc.go's #cgo pkg-config: jemalloc resolves to -ljemalloc; removing libjemalloc.so in the builder (Dockerfile:16) correctly forces static linking, consistent with the stated rationale.
• docker-compose.yml's healthcheck swap to juno --help and the golang.org/x/net v0.58.0 → v0.60.0 bump (indirect-only, via grpc, hashes match between go.mod/go.sum) are unchanged from previous rounds and remain fine.

No inline comments this round — the only blocker from prior reviews has been addressed and nothing new surfaced.
· branch: chore/docker-image-hardening

[github-actions]

Claude finished @brbrr's task in 6m 48s —— View job

---

Review complete

• Gather context (PR diff vs origin/main, changed files)
• Review Dockerfile changes
• Review Makefile changes
• Review docker/apko.yaml and docker/apko.lock.json
• Review docker-compose.yml changes
• Review go.mod/go.sum changes
• Post findings

Summary

The Docker/build-tooling files (Dockerfile, Makefile, docker/apko.yaml, docker/apko.lock.json, docker-compose.yml) are unchanged since the last review round (commit d066578d2) — the only new commit on the branch is a merge of main that brings in unrelated dependency bumps (bits-and-blooms/bitset, fxamacker/cbor, libp2p-pubsub, docs npm bumps), not touched by this PR's own diff. No new issues found in this pass.

This PR only touches build tooling and an indirect Go dependency bump — nothing under consensus/, sync/, p2p/, RPC handlers, or other Go runtime code, so the correctness/concurrency/protocol-path review focus doesn't apply here.

Carried-over blocker — already resolved

• The python3 -c manifest-parsing step in the base stage (Dockerfile:58-73) now explicitly installs python3 before using it, fixing the build failure flagged in the prior round. CI's container-security job builds this stage successfully, confirming the fix.

Nits (non-blocking, previously raised, still open by design)

• APKO_VERSION=v1.4.11 is pinned independently in both Dockerfile:53 and Makefile:126 with "keep in sync" comments rather than a single source of truth. Low risk since both already cross-reference each other.
• The final scratch image has no USER/accounts: directive, so the container still runs as root. The PR description documents this as an intentional trade-off (root-owned data volumes from earlier images), so not re-flagged as new.

Verified

• docker/apko.yaml / docker/apko.lock.json package sets, versions and architectures are consistent for x86_64 and aarch64 (glibc-2.44, ld-linux-2.44, libgcc, ca-certificates-bundle, tzdata, wolfi-baselayout).
• Dockerfile:16's removal of libjemalloc.so correctly forces -ljemalloc (via jemalloc/jemalloc.go's #cgo pkg-config: jemalloc) to link the static archive, matching the stated rationale for dropping the runtime jemalloc/libstdc++ dependency.
• docker-compose.yml's healthcheck swap to juno --help and the golang.org/x/net v0.58.0 → v0.60.0 bump (indirect-only, via grpc; hash and version match between go.mod/go.sum) are unchanged from previous rounds and remain fine.

No inline comments this round — everything previously flagged as a blocker has been addressed, and nothing new surfaced since the last review.
· branch: chore/docker-image-hardening

[thiagodeev]

The only caveat is that we no longer have the image updated on each build, but I think that is a feature, not a bug, right?

[github-actions]

Claude finished @brbrr's task in 5m 6s —— View job

---

Review complete

• Gather context (diff vs origin/main, changed files)
• Review new GitHub Actions workflow changes (publish debian + distroless variant)
• Review Dockerfile/Makefile/apko files (re-check for changes since last review)
• Review docs/docs/running-juno.md changes
• Post findings

Summary

The new commit (8c3be8879) changes strategy in response to the earlier "breaking change" concerns: instead of making the shell-free Wolfi image the only published image, it now publishes both variants —

• nethermind/juno:<tag> — the old Debian-based image with a shell, built from the new debian target (Dockerfile:50-64), kept as default so existing shell-based deployments (deploy-and-test.yaml, k8s workloads using sh -c) are unaffected.
• nethermind/juno:<tag>-distroless — the hardened, shell-free Wolfi/apko image from the distroless target (renamed from final), opt-in via tag suffix.

This is a reasonable way to de-risk the rollout. docker-image-build-push.yml, prepare-draft-release.yml, and release-published.yml consistently add the distroless build/push/tag-as-latest steps alongside the existing debian ones, and docs/docs/running-juno.md documents the new tag and the local --target debian build option. No remaining references to the old final stage name anywhere in the repo.

All blockers from prior review rounds (missing python3, brittle sed-based manifest parsing, USER root) were already resolved in earlier commits and are unchanged here. This PR only touches build tooling and CI workflows — no changes under consensus/, sync/, p2p/, RPC handlers, or other Go runtime code.

Nit (posted inline on Dockerfile:50)

• The new debian stage still pulls FROM debian:bookworm-slim as a floating tag, unlike the fully pinned/reproducible distroless path (digest-pinned apko builder image, docker/apko.lock.json). Since this is now a long-lived, separately-published compatibility image rather than a stepping stone, consider pinning it to a digest too. Not a blocker — it carries forward the pre-existing behavior from before this PR.

Also noted (non-blocking, not flagged inline)

• distroless carries the new OCI labels (org.opencontainers.image.*) and SSL_CERT_FILE; debian doesn't get either. Cosmetic only — Debian's own ca-certificates package already covers the default cert path.
• build_and_push_docker_image_amd/_arm jobs now invoke docker buildx build twice per platform (once per target), both sharing the same builder stage. Likely cached fine by buildx's per-job builder instance, but worth a glance if CI build times increase noticeably.
  · branch: chore/docker-image-hardening