From dd036b5ada8178e5123439b95d9716379537b8c7 Mon Sep 17 00:00:00 2001 From: Wei Date: Fri, 10 Jul 2026 11:17:47 +0800 Subject: [PATCH] feat(gcp-cvm): add Stage C (publish to GCP) + document the 3-stage pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The build produced a qcow2 but stopped short of a GCP image. Add the "last mile": - publish-gcp-image.sh (Stage C): qemu-img convert qcow2 -> raw -> sparse oldgnu tar.gz -> gsutil cp -> gcloud compute images create with UEFI_COMPATIBLE,GVNIC,SEV_CAPABLE,TDX_CAPABLE. Parameterized (GCS_BUCKET/GCP_PROJECT/GUEST_OS_FEATURES, defaults gs://tapp-image + g-devops); refuses to clobber an existing image name. Kept separate from the build because it needs gcloud/gsutil auth + network upload. - build-gcp-tapp.sh: opt-in Stage C via PUBLISH_AS= (runs publish after Stage B); also fix the now-stale tapp-data-provision.service Description (format-or-adopt, not "carve"). - Docs: README pipeline overview (Stage 0 base prep / A / B / optional smoke test / C), publish-gcp-image.sh in the file table, PUBLISH_AS env var, a "Publish to GCP (Stage C)" section; main doc §14 (Stage C steps + stages-at-a-glance). Co-Authored-By: Claude Opus 4.8 (1M context) --- gcp-cvm/README.md | 27 +++++++++++- gcp-cvm/build-gcp-tapp.sh | 13 +++++- gcp-cvm/cryptpilot-gcp-boot-fix.md | 28 +++++++++++++ gcp-cvm/publish-gcp-image.sh | 67 ++++++++++++++++++++++++++++++ 4 files changed, 132 insertions(+), 3 deletions(-) create mode 100755 gcp-cvm/publish-gcp-image.sh diff --git a/gcp-cvm/README.md b/gcp-cvm/README.md index ec8b478..6aa1c7c 100644 --- a/gcp-cvm/README.md +++ b/gcp-cvm/README.md @@ -6,16 +6,24 @@ Build a bootable, measurable, remotely-attestable, security-hardened cryptpilot | File | Description | |---|---| | `cryptpilot-gcp-boot-fix.md` | **Main doc**: root-cause analysis + fixes + full SOP (§9) + security-hardening audit (§11) + convert issues for Alibaba Cloud (§7) | -| `build-gcp-tapp.sh` | **One-shot full chain**: bare image → final gcp-tapp (stage A installs app/docker/SGX/DNS + hardening / stage B kernel + convert + ESP) | +| `build-gcp-tapp.sh` | **One-shot full chain**: base image → final gcp-tapp (Stage A app/docker/SGX/DNS + hardening + /data + Sysbox / Stage B kernel + convert + ESP / opt-in Stage C publish via `PUBLISH_AS=`) | | `prepare-gcp-tapp.sh` | Stage B only (when a base already exists): fix A + DNS (guestfish) + nbd reset + convert + fix B | +| `publish-gcp-image.sh` | **Stage C**: publish a built qcow2 to GCP — `qemu-img` raw → oldgnu sparse `tar.gz` → `gsutil` → `gcloud compute images create` (confidential guest-os-features). Needs gcloud/gsutil auth | | `fix-esp-grub.sh` | Sync the ESP grub only (fix B, can be run standalone against an already-converted image) | | `test/boot-smoke-test.sh` | **Local boot smoke test**: boots a converted image under QEMU/OVMF (no real GCP CVM needed) and checks the boot chain reaches multi-user / tapp-server | | `config_dir/` | cryptpilot convert config (`fde.toml`, `rw_overlay="ram"`) | | `cryptpilot-fde_0.7.0_amd64.deb` | FDE **runtime installed into the target image**. **Binary, gitignored**, must be placed locally in this directory (see Prerequisites) | -> The artifact `gcp-tapp.qcow2` (~6G, converted / verity-sealed / hardened) is the **output** of `build-gcp-tapp.sh` and is not committed (gitignored). +> The artifact `gcp-tapp.qcow2` (~4.5G, converted / verity-sealed / hardened) is the **output** of `build-gcp-tapp.sh` and is not committed (gitignored). > The same applies to `cryptpilot-fde_*.deb` and the tapp-server binary: the deb must be placed locally in this directory; tapp-server is pulled by default from GitHub release v0.1.0 (see below). +## Pipeline (stages) +- **Stage 0 — base prep** *(one-time, reused across builds)*: official Ubuntu 24.04 cloud image → resize to 20 GiB + gVNIC driver → base qcow2. See `cryptpilot-gcp-boot-fix.md` §0. This is the input to Stage A, not part of `build-gcp-tapp.sh`. +- **Stage A** (`build-gcp-tapp.sh`): provision app / docker / SGX / DNS, security hardening, Sysbox, and the `/data` + `br_netfilter` bakes. +- **Stage B** (`prepare-gcp-tapp.sh`, invoked by A): install the gcp kernel → fix A → `cryptpilot-convert` → sync ESP (fix B). +- *(optional)* local boot smoke test (`test/boot-smoke-test.sh`). +- **Stage C** (`publish-gcp-image.sh`): raw → sparse `tar.gz` → GCS → `gcloud compute images create`. Run standalone, or from `build-gcp-tapp.sh` via `PUBLISH_AS=`. + ## Prerequisites - **Conversion host = Anolis / Alibaba Cloud Linux 3 (al8).** `cryptpilot-convert` is only packaged for al8; a plain Ubuntu/macOS host cannot run it. - Install the host tool and dependencies (run the build as **root**): @@ -46,6 +54,7 @@ KBS_URLS='"http://:9091", "http://:9091"' \ - `KBS_URLS` — KBS node URLs for `[kbs] node_urls`, comma-separated and quoted as shown. - tapp-server is downloaded by default from GitHub v0.1.0 (includes the guest-components `8d71a3b4` fix, RTMR OK); if you have it locally, set `TAPP_SERVER_BIN=`. - Storage / Sysbox knobs: `DATA_ROOT` (docker data-root, default `/data/docker`), `CONTAINERD_ROOT` (default `/data/containerd`), `DOCKER_VERSION` (default `5:27.5.1-…noble`; empty = repo default), `ENABLE_SYSBOX` / `SYSBOX_VERSION` (default `0.7.0`). +- Publish (Stage C, opt-in): `PUBLISH_AS=` publishes the built image to GCP after the build (see [Publish to GCP](#publish-to-gcp-stage-c)); `GCS_BUCKET` / `GCP_PROJECT` / `GUEST_OS_FEATURES` pass through. - Other environment variables: `DNS_FALLBACK` `PURGE_KERNEL` `CONFIG_DIR` `FDE_PACKAGE` `ROOTFS_MODE` `IN_PLACE` `INSTALL_KERNEL` `NBD_RESET` (see the top of the script). ## Persistent data disk (`/data`) — always configured @@ -86,6 +95,20 @@ It boots the image under QEMU/OVMF (UEFI) in the `qemux/qemu` container — usin This validates everything except the TDX-specific bits (RTMR extend, remote attestation), which require real hardware — so it is a fast pre-flight check, not a replacement for on-hardware testing. Tunables: `MAX` (timeout seconds), `RAM_SIZE`, `CPU_CORES`. + +## Publish to GCP (Stage C) +A qcow2 can't be uploaded to GCP directly — it must become a `disk.raw` inside a sparse `oldgnu` tarball, then a GCP image. `publish-gcp-image.sh` does the four steps (`qemu-img convert` → `tar --format=oldgnu -Szcf` → `gsutil cp` → `gcloud compute images create` with `UEFI_COMPATIBLE,GVNIC,SEV_CAPABLE,TDX_CAPABLE`): +```bash +gcloud auth login # gcloud + gsutil must be authenticated with write access to the bucket/project +./publish-gcp-image.sh /path/og-tdx-dev.qcow2 og-tdx-dev +./publish-gcp-image.sh /path/og-tdx.qcow2 og-tdx +``` +Defaults `GCS_BUCKET=gs://tapp-image`, `GCP_PROJECT=g-devops`, `GUEST_OS_FEATURES=UEFI_COMPATIBLE,GVNIC,SEV_CAPABLE,TDX_CAPABLE` (all overridable). It refuses to clobber an existing image name (delete it, or publish under a new name). Or fold it into the build as an opt-in final stage: +```bash +PUBLISH_AS=og-tdx-dev ENABLE_SYSBOX=1 OWNER_ADDRESS=0x... KBS_URLS='...' ./build-gcp-tapp.sh base.qcow2 og-tdx-dev.qcow2 +``` +Create a confidential instance from the published image with `--image= --image-project=g-devops --confidential-compute-type=TDX`. + ## Three core fixes (all required) - **Fix A**: before convert, point the `/boot/vmlinuz` symlink at the gcp kernel → the cryptpilot stack goes into the correct initrd (fixes read-only / RTMR / verity). - **Fix B**: after convert, sync the boot-partition grub.cfg + modules to the ESP (fixes the boot crash bli.mod / vmlinuz not found). diff --git a/gcp-cvm/build-gcp-tapp.sh b/gcp-cvm/build-gcp-tapp.sh index 93d2b2a..211bced 100755 --- a/gcp-cvm/build-gcp-tapp.sh +++ b/gcp-cvm/build-gcp-tapp.sh @@ -45,6 +45,10 @@ export FDE_PACKAGE="${FDE_PACKAGE:-$HERE/cryptpilot-fde_0.7.0_amd64.deb}" export ROOTFS_MODE="${ROOTFS_MODE:---rootfs-no-encryption}" export PURGE_KERNEL="${PURGE_KERNEL:-}" # NOTE: convert needs at least one *-generic kernel left in the image; do not purge them all export INSTALL_KERNEL=1 +# Stage C (opt-in): if set, publish $OUT to GCP as this image name after the build (raw -> tar.gz -> +# gsutil -> gcloud images create). Empty = build only. Needs gcloud/gsutil auth; see publish-gcp-image.sh +# (GCS_BUCKET / GCP_PROJECT / GUEST_OS_FEATURES are passed through to it). +PUBLISH_AS="${PUBLISH_AS:-}" # ==================== IN="${1:?usage: $0 }" @@ -225,7 +229,7 @@ PROVSH chmod 0755 /usr/local/sbin/tapp-data-provision.sh cat > /etc/systemd/system/tapp-data-provision.service <<'PROVUNIT' [Unit] -Description=Auto-provision /data (carve from boot-disk free space if no tapp-data fs exists) +Description=Auto-provision /data (format a blank data disk, or adopt an existing one, as tapp-data) After=systemd-udev-settle.service Wants=systemd-udev-settle.service Before=data.mount @@ -381,3 +385,10 @@ IN_PLACE=1 "$HERE/prepare-gcp-tapp.sh" "$IN" "$OUT" echo "" echo "[done] final image: $OUT" echo "(note: $IN was modified in place; re-download the original base image if you need it again)" + +# ---- stage C (opt-in): publish $OUT to GCP as image $PUBLISH_AS ---- +if [ -n "$PUBLISH_AS" ]; then + [ -x "$HERE/publish-gcp-image.sh" ] || { echo "PUBLISH_AS set but publish-gcp-image.sh not found/executable in $HERE" >&2; exit 1; } + echo "==> [C] publish-gcp-image.sh: $OUT -> GCP image '$PUBLISH_AS'" + "$HERE/publish-gcp-image.sh" "$OUT" "$PUBLISH_AS" +fi diff --git a/gcp-cvm/cryptpilot-gcp-boot-fix.md b/gcp-cvm/cryptpilot-gcp-boot-fix.md index 1ab6585..28573aa 100644 --- a/gcp-cvm/cryptpilot-gcp-boot-fix.md +++ b/gcp-cvm/cryptpilot-gcp-boot-fix.md @@ -459,3 +459,31 @@ The rootfs writable overlay is `rw_overlay = "ram"` (zram) — **anything writte - **Sysbox data store** (`sysbox-mgr --data-root`, holds inner-container images) is also moved to `/data/sysbox` via a drop-in that preserves the vendor `ExecStart`. - **Kernel**: idmapped mounts (gcp kernel ≥5.12) are required; **6.17.0-*-gcp verified on hardware** (`docker run --runtime=sysbox-runc alpine cat /proc/self/uid_map` → `0 100000 65536`). - **`br_netfilter` must be loaded** (baked via `/etc/modules-load.d/`, unconditional). Docker 28's `icc=false` bridges — created by the 0g-sandbox runner (`INTER_SANDBOX_NETWORK_ENABLED=false`) — hard-require `/proc/sys/net/bridge/bridge-nf-call-iptables`, which exists only once `br_netfilter` is loaded. On a fresh CVM without it the runner crash-loops (`bridge-nf-call-iptables: no such file`); a node where it happened to be loaded worked, masking the gap. + +## 14. Stage C: publish the image to GCP + +A built qcow2 (§9 output) cannot be uploaded to GCP directly — GCP wants a raw disk named exactly `disk.raw` inside a **sparse `oldgnu`-format tarball**, imported as an image with the confidential guest-os-features. `gcp-cvm/publish-gcp-image.sh ` runs the four steps: + +```bash +# [C1] qcow2 -> raw +qemu-img convert -p -f qcow2 -O raw og-tdx.qcow2 disk.raw +# [C2] raw -> sparse oldgnu tarball (member MUST be exactly "disk.raw") +tar --format=oldgnu -Szcf og-tdx.tar.gz -C disk.raw +# [C3] upload +gsutil cp og-tdx.tar.gz gs://tapp-image/ +# [C4] create the GCP image (confidential features) +gcloud compute images create og-tdx --project=g-devops \ + --source-uri=gs://tapp-image/og-tdx.tar.gz \ + --guest-os-features=UEFI_COMPATIBLE,GVNIC,SEV_CAPABLE,TDX_CAPABLE +``` + +Notes: +- **Auth**: needs `gcloud auth login` and write access to the bucket/project. This is why Stage C is separate from the build (which needs no cloud creds) — run it after §9, or fold it in with `PUBLISH_AS=` on `build-gcp-tapp.sh`. +- **`--format=oldgnu -S`**: GCP's importer requires the GNU-old tar format; `-S` keeps the (mostly-empty) 20 GiB raw sparse so the tarball stays small (~few GB). +- **`-C disk.raw`**: the archive member must be the bare name `disk.raw`, not a path. +- **Guest-OS features**: `UEFI_COMPATIBLE` (cryptpilot boots via UEFI/ESP), `GVNIC` (network driver), `SEV_CAPABLE`+`TDX_CAPABLE` (usable as an AMD-SEV or Intel-TDX confidential VM). +- The script refuses to overwrite an existing image name (delete it first, or publish a versioned name, e.g. `og-tdx-YYYYMMDD`). +- Boot a confidential instance from it: `gcloud compute instances create … --image=og-tdx --image-project=g-devops --confidential-compute-type=TDX --maintenance-policy=TERMINATE`. + +### Stages at a glance +**Stage 0** (§0, one-time base prep: official Ubuntu → resize 20 GiB + gVNIC) → **Stage A** (provision + harden + Sysbox + /data/br_netfilter) → **Stage B** (gcp kernel + convert + ESP) → *(optional local boot smoke test, README)* → **Stage C** (publish to GCP). diff --git a/gcp-cvm/publish-gcp-image.sh b/gcp-cvm/publish-gcp-image.sh new file mode 100755 index 0000000..7e797ba --- /dev/null +++ b/gcp-cvm/publish-gcp-image.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# publish-gcp-image.sh +# +# Stage C: turn a built cryptpilot tapp qcow2 into a bootable GCP image. Four steps: +# [C1] qemu-img convert qcow2 -> raw disk.raw +# [C2] tar (oldgnu, sparse) -> .tar.gz (GCP requires an archive whose single member is +# exactly "disk.raw") +# [C3] gsutil cp the tarball to the GCS bucket +# [C4] gcloud compute images create with the confidential-VM guest-os-features +# +# Requires: qemu-img, tar, gsutil, gcloud (authenticated: `gcloud auth login`), and write access to +# the target GCS bucket / project. This step is separate from the image build because it needs cloud +# credentials and network upload; run it after build-gcp-tapp.sh (or via that script's opt-in Stage C). +# +# Usage: +# ./publish-gcp-image.sh /root/og-tdx.qcow2 og-tdx +# GCS_BUCKET=gs://tapp-image GCP_PROJECT=g-devops ./publish-gcp-image.sh og-tdx-dev.qcow2 og-tdx-dev + +set -euo pipefail + +IMG="${1:?usage: $0 }" +NAME="${2:?usage: $0 }" +[ -f "$IMG" ] || { echo "image not found: $IMG" >&2; exit 1; } + +# ===== Tunables ===== +GCS_BUCKET="${GCS_BUCKET:-gs://tapp-image}" +GCP_PROJECT="${GCP_PROJECT:-g-devops}" +GUEST_OS_FEATURES="${GUEST_OS_FEATURES:-UEFI_COMPATIBLE,GVNIC,SEV_CAPABLE,TDX_CAPABLE}" +WORKDIR="${WORKDIR:-$(cd "$(dirname "$IMG")" && pwd)}" # where disk.raw / tarball are staged +KEEP_TARBALL="${KEEP_TARBALL:-0}" # 1 = keep the local .tar.gz after upload +# ==================== + +for t in qemu-img tar gsutil gcloud; do command -v "$t" >/dev/null || { echo "missing tool: $t" >&2; exit 1; }; done + +RAW="$WORKDIR/disk.raw" +TAR="$WORKDIR/$NAME.tar.gz" + +# refuse to clobber an existing image name (gcloud would error anyway; give a clearer hint up front) +if gcloud compute images describe "$NAME" --project="$GCP_PROJECT" >/dev/null 2>&1; then + echo "GCP image '$NAME' already exists in project $GCP_PROJECT." >&2 + echo " delete it first: gcloud compute images delete $NAME --project=$GCP_PROJECT" >&2 + echo " or publish under a new name (e.g. ${NAME}-$(date +%Y%m%d) — pass a different )." >&2 + exit 1 +fi + +cleanup() { rm -f "$RAW"; [ "$KEEP_TARBALL" = 1 ] || rm -f "$TAR"; } +trap cleanup EXIT + +echo "==> [C1] qemu-img convert qcow2 -> raw ($RAW)" +qemu-img convert -p -f qcow2 -O raw "$IMG" "$RAW" + +echo "==> [C2] tar (oldgnu, sparse) -> $TAR" +# -C so the archive member is exactly "disk.raw" (GCP requirement), -S to keep the sparse raw small +tar --format=oldgnu -Szcf "$TAR" -C "$WORKDIR" disk.raw + +echo "==> [C3] upload -> $GCS_BUCKET/" +gsutil cp "$TAR" "$GCS_BUCKET/" + +echo "==> [C4] create GCP image '$NAME' (project $GCP_PROJECT, features $GUEST_OS_FEATURES)" +gcloud compute images create "$NAME" \ + --project="$GCP_PROJECT" \ + --source-uri="$GCS_BUCKET/$(basename "$TAR")" \ + --guest-os-features="$GUEST_OS_FEATURES" + +echo "" +echo "[done] GCP image: $NAME (project $GCP_PROJECT)" +echo " create an instance with: --image=$NAME --image-project=$GCP_PROJECT --confidential-compute-type=TDX"