feat(gcp-cvm): add GCP confidential image build kit

[Wilbert957]

Build a hardened, measured, remotely-attestable GCP (Intel TDX) confidential image for 0g-tapp from a stock Ubuntu 24.04 cloud image, using cryptpilot for measured FDE.

What's here

• gcp-cvm/build-gcp-tapp.sh — one-command pipeline: base provisioning (tapp-server + Docker + Intel SGX/libtdx-attest + config + DNS) → kernel swap (generic → gcp, for RTMR extend) → cryptpilot-convert → ESP grub sync → optional security hardening. Toggle with HARDEN=1|0.
• gcp-cvm/prepare-gcp-tapp.sh / gcp-cvm/fix-esp-grub.sh — stage-B-only (kernel + convert + ESP) and ESP-only helpers, reused by the main pipeline.
• gcp-cvm/cryptpilot-gcp-boot-fix.md — root-cause analysis + full reproducible SOP + integrity notes, covering the four problems we hit (boot crash from GCP's dual grub.cfg, read-only rootfs / RTMR-not-extended, runtime RTMR extend failure, broken DNS) and the convert-side issues to confirm with the cryptpilot maintainers.
• gcp-cvm/README.md, gcp-cvm/config_dir/fde.toml, root README.md pointer, .gitignore for *.deb/*.qcow2.

Two image variants (same base, same pipeline)

	hardened (HARDEN=1)	dev (HARDEN=0)
Purpose	production	debugging
ssh / cloud-init / google agents / snapd / …	removed	kept
google-guest-agent (GCP SSH key injection)	omitted by design → not SSH-reachable	installed → SSH works
kernel / gve / cryptpilot / tapp-server v0.1.0	identical	identical

google-guest-agent is a back-door-class component (it can push changes into the instance from outside the measured app), so the hardened image is intentionally not SSH-reachable; the dev image reinstalls it (+ a 169.254.169.254 metadata.google.internal /etc/hosts entry, needed because the build pins resolv.conf to public DNS) to restore SSH.

§0 reproducibility note

The regression was run from the truly-official Ubuntu cloud image, which revealed that the previously-used temp-fixed.qcow2 base was not a pristine "official + gVNIC" image — it also carried (1) the root partition resized to 20 GiB and (2) google-guest-agent. Both are now documented in §0 and handled by the build, so the image is reproducible from the official image alone.

Companion fix

The cryptpilot show-reference-value fix for never-booted images (empty grubenv / missing saved_entry) is openanolis/cryptpilot#128.

Binaries are gitignored; tapp-server is pulled from release v0.1.0.

🤖 Generated with Claude Code

[Wilbert957]

✅ Reviewed. Pure shell scripts + docs, no Rust code changes — no compilation impact.

• build-gcp-tapp.sh / prepare-gcp-tapp.sh / fix-esp-grub.sh: well-structured, proper set -euo pipefail, env-var overrides for all paths.
• Security hardening (§11): thorough purge of ssh/cloud-init/google-guest-agent/snapd/etc. Masking getty + MAC-agnostic netplan is correct.
• .gitignore correctly excludes *.qcow2 / *.deb / disk.raw.
• cryptpilot-gcp-boot-fix.md: detailed root-cause analysis + SOP.

One suggestion: cryptpilot-fde_0.7.0_amd64.deb is referenced but gitignored — consider adding a sha256sum check in the build script to verify the deb when present. Not blocking.

LGTM.

[Wilbert957]

Read through all three scripts + fde.toml + the smoke test. Agree with the LGTM — well-structured defensive shell (consistent set -euo pipefail, env overrides, required-arg checks, trap cleanup, guestfish layout asserts), clean two-variant design, and the QEMU/OVMF smoke test is a genuinely useful pre-flight. A few notes:

Worth confirming before treating any output as a production image

• config_dir/fde.toml hardcodes a literal passphrase:

  [data.encrypt.exec]
  command = "echo"
  args = ["-n", "test-passphrase"]

  Even if this data volume is inert here ([data] integrity = false + build uses --rootfs-no-encryption, so rootfs is verity-only), baking test-passphrase into the image-build config is a smell. Can we confirm it's never exercised, or swap it for a real key provider for prod?

Non-blocking / follow-up

• No checksum on downloaded inputs: tapp-server (wget) and FDE_PACKAGE (deb). The measured image catches post-build tampering, but a sha256sum on both inputs is cheap defense-in-depth (extends the deb suggestion already raised).
• build-gcp-tapp.sh mutates $IN in place (stage A operates on the input directly), whereas prepare-gcp-tapp.sh defaults to a protective copy. The trailing echo warns about it, but given the (in, out) signature this is a footgun — consider copying first or announcing the in-place destruction up front.
• Partition layout is hardcoded (base sda1; post-convert sda15=ESP / sda16=boot). Fine for GCP Ubuntu images, and the is-file/is-dir asserts fail loudly on mismatch rather than writing the wrong partition. OK as-is.

Naming nit (in favor): gcp-cvm/ is correct here — this genuinely builds a GCP-specific image, distinct from the verification side which is platform-agnostic. Don't let anyone "unify" it away.

[Wilbert957]

Docker version pin + Sysbox compatibility

Empirically tested against wei-tapp2 (Ubuntu 24.04, kernel 6.17, GCP). Docker CE from the official apt repo currently pulls 29.6.0, which breaks Sysbox 0.7.0 (and 0.6.5, this PR's default):

docker run --rm --runtime=sysbox-runc alpine echo hi
→ OCI runtime create failed: namespace {"time" ""} does not exist

Two independent Docker 29 additions Sysbox hasn't caught up with:

1. Docker 29 emits the Linux 5.6 time namespace in the OCI spec (namespaces: [{type: "time"}]). sysbox-runc parses it as an unknown namespace type and refuses to create the task.
2. Docker 29 also calls sysbox-runc features at startup to discover runtime capabilities — that subcommand doesn't exist in sysbox-runc, so dockerd logs No help topic for 'features' on every restart (non-fatal but noisy, and it means Docker's capability negotiation is falling back to a legacy path).

Downgrading Docker to 27.5.1 (last 27.x, still in the same apt repo) resolves both:

docker run --rm --runtime=sysbox-runc alpine cat /proc/self/uid_map
→ 0    362144    65536      # user-ns remap actually in effect

Nestybox's compatibility matrix officially covers Docker 20.10–27.x. There's no released Sysbox that supports Docker 28+ yet.

Suggested change to build-gcp-tapp.sh

Pin Docker to a 27.x version in the apt install line, e.g.:

DOCKER_VERSION="${DOCKER_VERSION:-5:27.5.1-1~ubuntu.24.04~noble}"
apt-get install -y libtdx-attest   docker-ce="$DOCKER_VERSION" docker-ce-cli="$DOCKER_VERSION"   containerd.io docker-buildx-plugin docker-compose-plugin

Otherwise ENABLE_SYSBOX=1 produces an image where sysbox-runc is registered but no container can actually launch with it. The failure is silent from the image-build side — it only surfaces the first time an operator tries to run a hostile-multi-tenant workload.

Also worth bumping the default SYSBOX_VERSION from 0.6.5 to 0.7.0 (2026-06-02). Same Docker constraints, but 0.7.0 has fixes we've been running against.

Happy to open a small companion PR pinning the version once you confirm the direction.

[Wilbert957]

Follow-up after hitting this for real on a node: the /data pinning is incomplete — it pins docker's data-root but not containerd's root, so with the current docker-ce default the image layers still land on the rootfs.

Evidence

On a plain docker-ce install (noble, no daemon.json), docker info reports:

Storage Driver: overlayfs
 driver-type: io.containerd.snapshotter.v1

i.e. current docker-ce defaults to the containerd image store. In that mode image layers are managed by containerd and stored under containerd's root (/var/lib/containerd), not under docker's data-root. data-root only moves docker's own metadata (~2G: volumes, buildkit).

Observed on a running node with the sandbox stack: data-root=/data/docker held ~2G, but /var/lib/containerd held 8.7G of image layers on the rootfs — filling / to 65%. Exactly the "RAM rootfs overlay OOM / ephemeral" failure this PR's README warns against. So as written, the pin doesn't actually keep images off the root.

Fix — also pin containerd's root in the build

Alongside the docker daemon.json data-root, set containerd's root to /data and make containerd (which docker depends on) wait for /data too:

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's|^\(\s*\)root = .*|root = "/data/containerd"|' /etc/containerd/config.toml
mkdir -p /etc/systemd/system/containerd.service.d
printf '[Unit]\nRequiresMountsFor=/data\n' > /etc/systemd/system/containerd.service.d/10-data-mount.conf

(Bake it into the image at build time, as this PR does for the docker side.)

Secondary: the whole /data pin is gated behind ENABLE_SYSBOX=1

The "keep container data off the RAM rootfs" requirement is independent of sysbox — an image built with ENABLE_SYSBOX=0 gets no data-root/containerd pin and will pile images onto the root. Suggest making the /data pinning (docker data-root + containerd root + RequiresMountsFor=/data + fstab) unconditional, and keeping only the sysbox-runc runtime registration under the ENABLE_SYSBOX flag.

Note

I worked around this on the running node with a bind mount (/data/containerd → /var/lib/containerd) since I couldn't rebuild the image there, but the correct place is here in the build — the containerd root should be customized during image creation, same as docker's.

[Wilbert957]

Follow-ups in 58a7307 (from further GCP testing + the 0g-sandbox deploy)

1. /data now adopts an existing (migrated) data disk — no SSH, no relabel.
tapp-data-provision picks the single non-boot disk and: blank → mkfs.ext4 -L tapp-data; already ext4 (e.g. a chain-data disk moved from another VM) → adopt via e2label tapp-data, never reformat (data preserved). Guarded to real disks only, never the boot/partitioned disk, refuses to guess on 0 or >1 candidates. So attaching any single data disk — brand-new or carrying data — just works.

2. br_netfilter auto-loaded (/etc/modules-load.d/, unconditional).
Surfaced deploying 0g-sandbox to a fresh CVM (finding originally posted on 0g-sandbox#51 — it belongs here in the CVM build). Docker 28's icc=false bridges (created by the sandbox runner) hard-require /proc/sys/net/bridge/bridge-nf-call-iptables, which exists only once br_netfilter is loaded. This minimal image doesn't autoload it → a fresh CVM crash-loops the runner; wei-tapp2 only worked because the module happened to be loaded. Now persisted so every CVM has it at boot.

Verified on GCP (kernel 6.17-gcp): br_netfilter.conf present, provision adopts/formats correctly, prior fixes intact (fuse3, docker 27.5.1, nofail + device-timeout=60s).

Deploy-side doc updated in the companion PR 0gfoundation/0g-tee-full-node#1 (drops the fragile manual docker data-root step, adds the auto data-disk flow).