feat(daytona): upgrade to v0.189 + Sysbox host-boundary isolation

[Wilbert957]

Summary

• Bump 4 daytona images to v0.189.0-amd64 (from pinned Feb 2026 sha)
• New init container grant-admin-perms grants the daytona-admin API key the full v0.189 permission enum (v0.189 introduces api_key.permissions; existing admin keys carry migration default {} → 403 on writes)
• Client.ListSandboxes handles the v0.189 {items, nextCursor} paginated envelope. Silent-empty regression was blocking /api/sandbox_list, owner-scoped /api/sandbox, /api/sessions, /api/archive-all, and archiveRunningOnShutdown
• Security hardening (see dedicated section below): outer-layer Sysbox on the runner + native cross-sandbox network isolation. Compose diff: runtime: sysbox-runc on runner + INTER_SANDBOX_NETWORK_ENABLED=false on runner env; the old runner-firewall iptables sidecar from PR fix(runner): enforce cross-sandbox network isolation via firewall sidecar #45 is removed (its DOCKER-USER rule silently no-ops on hosts without br_netfilter)

Host-side dependency (Sysbox install + sysbox-runc registration + Docker version pin) is managed by 0g-tapp#5 — see companion review comment on Docker 27 pin + Sysbox 0.7 compat.

---

Security hardening — two questions

Multi-tenant sandboxes on one wei-tapp host raise two separate isolation questions. This PR addresses both.

Q1. Can a user in one sandbox get root on the host?

Status before this PR: Yes, trivially. daytona-runner v0.189 hard-codes HostConfig.Privileged = true for every CPU sandbox it spawns (apps/runner/pkg/docker/container_configs.go:205). Combined with the sandbox image's passwordless sudo, that means:

• Container uid=0 is the host kernel's uid=0 (no user-namespace remap; standard runc shares the host kernel directly)
• All Linux capabilities are granted (CAP_SYS_ADMIN, CAP_SYS_MODULE, CAP_NET_ADMIN, …)
• AppArmor / SELinux is disabled per SecurityOpt: [label=disable]
• Full device access (/dev/* reachable)

Without any CVE, a tenant can:

• sudo mount --bind /path/on/host /mnt/exfil — read any host path
• sudo mknod /dev/sda b 8 0 && dd if=/dev/sda … — read raw host disks
• Any kernel-CVE PoC that assumes CAP_SYS_ADMIN

Fix: runtime: sysbox-runc on the runner service. The host dockerd starts the runner container under sysbox-runc, which puts the entire runner into a user namespace. Runner's inner DinD + every user sandbox it later creates all live inside that user-ns. Container uid=0 is mapped to host uid=362144 (a normal user).

Effect chain:

1. Tenant sandbox root does something privileged → succeeds inside the nested user-ns
2. If the tenant escapes their sandbox container into the runner (e.g., via a container-runtime CVE) → still user-ns uid=0 = host uid=362144
3. To escalate to host root, the tenant would have to break out of the user namespace itself, not out of the container. That's kernel enforcement code, hardened for a decade+

Concrete escape CVEs that stop working under this design: CVE-2019-5736 (runc), CVE-2022-0185 (fsconfig), CVE-2024-21626 (runc file-descriptor leak) — all require CAP_SYS_ADMIN on the initial user-ns, which the tenant never holds.

Residual risk (honest):

• The tenant is still root inside their user-ns, so they can create nested containers, bind-mount their own paths, run kernel-fuzzing tools — anything that stays inside the ns is unhindered
• Kernel bugs that require no capabilities at all (very rare, mostly theoretical) are unaffected by Sysbox
• Host disk pressure: a runaway sandbox can still fill /var/lib/docker; that's a quota problem, not an isolation one

Q2. Can a user in one sandbox reach another sandbox?

Split by attack vector:

Q2a — Network path (sandbox A ↔ sandbox B over IP)

Status before this PR: On the Feb 2026 stack, sandboxes shared the runner's default docker0 bridge with enable_icc: true. PR #45 tried to fix it by inserting iptables -I DOCKER-USER 1 -i docker0 -o docker0 -j DROP via a sidecar. That rule only fires when the host has the br_netfilter kernel module loaded — default GCP Ubuntu 24.04 does not load it, so the rule was silently a no-op (verified on wei-tapp2: pkts=0 after 20+ minutes). Sandbox A could reach sandbox B on any port.

Fix: INTER_SANDBOX_NETWORK_ENABLED=false on the runner service (v0.189 native). daytona-runner now creates a dedicated bridge runner-bridge with com.docker.network.bridge.enable_icc: false, and spawns every sandbox on it. dockerd itself drops container-to-container packets on that bridge — no host kernel module needed. The old runner-firewall sidecar is removed.

Verified:

# From sandbox B (running python3 -m http.server 9000 on sandbox A):
curl http://<sandbox-A-ip>:9000/  → Connection timed out (HTTP 000, exit 28)
bash -c 'echo > /dev/tcp/<sandbox-A-ip>/9000'  → TCP-BLOCKED

Q2b — Filesystem / process path (sandbox A reads sandbox B's files or processes)

Status before this PR: Blocked by standard Docker container namespacing — each sandbox has its own PID, mount, IPC, user, UTS namespaces. Privileged: true grants full capabilities within the container's own namespaces; it does not let one container see into another container's private namespaces.

This PR does not change that behavior. What it verifies (and what a design review should check):

• /var/run/docker.sock is not mounted into user sandboxes — a compromised sandbox cannot use the Docker API to docker exec into a sibling. Confirmed: ls -la /var/run/docker.sock inside a running sandbox returns No such file or directory.
• Sandboxes don't share writable volumes with each other. Only per-sandbox volumes are mounted (verified via docker inspect on user containers).

Residual risk:

• If a future change accidentally mounts docker.sock into a sandbox (for e.g. dev-mode "run docker in your sandbox" feature), Q2b re-opens completely. Worth a linting check.

Summary matrix

Attack	Before PR #51	After PR #51	Enforced by
sandbox root → host root	Open	Blocked	Sysbox user-ns (host kernel)
sandbox A network → sandbox B network	Effectively open (br_netfilter not loaded)	Blocked	dockerd icc=false on runner-bridge
sandbox A fs / processes → sandbox B	Blocked	Blocked	Docker container namespaces (unchanged)
sandbox → spawn privileged sibling container via docker.sock	N/A (socket not mounted)	N/A	Daytona-runner default (unchanged)

Host-side prerequisite

sysbox-runc must be registered as a runtime in the host's /etc/docker/daemon.json, and the host must run Sysbox 0.7.0+ against Docker 27.x (Sysbox has not yet released a version compatible with Docker 28+ — see the 0g-tapp#5 review comment). If those aren't in place, docker compose up on the runner fails with unknown or invalid runtime name: sysbox-runc — safe-fail rather than silent-fall-back.

---

Test plan (end-to-end on wei-tapp2, a real testnet tapp deployment)

• fresh v0.189 install: grant-admin-perms skipped (idempotent guard), full lifecycle ok
• Feb 2026 → v0.189 upgrade with pre-existing sandboxes: pg_dump → change tags → docker compose up -d → migrations auto-run (65 → 111) → grant-admin-perms fires (UPDATE 1) → class → sandboxClass schema migration preserves data → old sandbox start/stop still work → new sandbox creates cleanly
• Rollback via pg_dump restore + revert tags → schema back to class, old sandbox intact
• ssh-access, toolbox on normal sandbox → 200; on sealed sandbox → 403
• owner-scoped list and admin sandbox_list both return correct count after pagination fix
• real chain settlement: 4 vouchers settled on-chain via settleFeesWithTEE, provider earnings updated
• stop→start preserves user data (/home/daytona/marker.txt survives)
• Q1 verified: runner container Runtime=sysbox-runc, uid_map 0 362144 65536; user sandbox uid_map also remapped; sudo mount --bind / inside sandbox mounts within nested user-ns only, no visibility into host /
• Q2a verified: curl http://<sandbox-A-ip>:9000/ from sandbox B → Connection timed out; raw bash /dev/tcp/<A>/9000 → TCP-BLOCKED
• Q2b verified: /var/run/docker.sock not present in sandbox; sandboxes have independent PID/mount namespaces
• sealed sandbox: image content digest resolved, SANDBOX_SEAL_ATTESTATION env injected, 0g-sealed:true label set, ssh-access + toolbox blocked at billing-proxy
• go build ./... + go test ./internal/daytona/... ./internal/proxy/...

Follow-up (out of scope, filed as PR comment)

Move to drop-DinD for Sysbox at the inner (user sandbox) boundary would additionally isolate user containers from the runner itself, but requires resolving the internal registry reachability question. This PR takes the outer-boundary approach because it needs no other topology changes.

🤖 Generated with Claude Code

[Wilbert957]

Follow-up out of scope: sandbox-runtime hardening

Recording the sandbox privilege-escalation surface so we don't lose it. Not fixing here — this PR is about the Daytona version bump + native cross-sandbox network isolation.

What's still there

The user container that Daytona spawns per sandbox is:

• Privileged: true (CPU sandboxes; hard-coded in daytona-runner v0.189 at apps/runner/pkg/docker/container_configs.go:205)
• SecurityOpt: [label=disable] (SELinux/AppArmor off)
• Runtime: runc (host kernel shared, no user-ns remap)
• Passwordless sudo inside → any workload can escalate to container root and use every capability

Anything a user runs inside a sandbox can mount --bind, load kernel modules, walk /dev/*, and reach the host kernel directly. Cross-sandbox network is now isolated (this PR), but host isolation isn't.

How Daytona says to fix it

Daytona already ships the hook: daytona-runner reads CONTAINER_RUNTIME env and hands it to Docker as hostConfig.Runtime (apps/runner/cmd/runner/config/config.go:33). Their docs point at sysbox-runc. Sysbox's user-ns remap turns "root in container" into "unprivileged user on host".

Why it doesn't drop in cleanly for us

Our compose runs runner: privileged: true — DinD. The nested dockerd doesn't inherit host runtimes, so CONTAINER_RUNTIME=sysbox-runc on the runner fails with "unknown or invalid runtime name". Daytona's own production model is drop-DinD: mount the host docker socket, remove privileged, let the host's dockerd (with sysbox-runc) create the sandbox containers.

I tried the drop-DinD swap end-to-end on wei-tapp2. Runner + host docker + Sysbox 0.7 all work individually — the wall is our internal registry:6000 service. Under DinD it was reachable because Daytona-runner mediated the pull from inside app-net; under drop-DinD, sandbox containers are on the host default bridge and can't resolve registry:6000 at all.

Two viable follow-up shapes (both are their own PR):

1. Publish registry on 127.0.0.1:6000 on the host and rewrite INTERNAL_REGISTRY_URL accordingly. Small compose diff, keeps INTER_SANDBOX_NETWORK_ENABLED=false native isolation.
2. Set CONTAINER_NETWORK=<app-net> on the runner so sandboxes attach to the compose network. Fixes DNS but loses the native isolation (they'd share a bridge with the rest of the stack).

Host side (Docker + Sysbox install + version pin) is 0g-tapp's concern — see companion comment on 0g-tapp#5. Filing this here so whoever picks up the sandbox side has the design context.

[Wilbert957]

Host-dependency finding: br_netfilter must be loaded on the CVM (new hard requirement from this PR)

Deploying this compose to a fresh GCP CVM (wei-tapp, 136.111.63.175) surfaced a runner crash-loop that a first deploy on wei-tapp2 didn't hit. Root cause is a host-provisioning gap that this PR turns from latent into fatal.

Symptom

runner container crash-loops (Restarting, all other services healthy). Inner DinD log:

level=error msg="Handler for POST /v1.51/networks/create returned error: cannot restrict
inter-container communication or run without the userland proxy:
stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory:
set environment variable DOCKER_IGNORE_BR_NETFILTER_ERROR=1 to ignore"
ERR Error creating Docker client wrapper error="failed to create runner-bridge network ..."

Root cause

The runner's inner dockerd is Docker 28, which hard-requires /proc/sys/net/bridge/bridge-nf-call-iptables to create a bridge with enable_icc=false. That sysctl only exists when the br_netfilter kernel module is loaded on the host.

This PR's INTER_SANDBOX_NETWORK_ENABLED=false makes daytona-runner create runner-bridge with icc disabled — so br_netfilter went from "optional" (old iptables sidecar path, PR #45) to a hard boot requirement for the runner. Ironically it's the same module PR #45's DOCKER-USER rule silently no-op'd without; now its absence is fatal instead of silent.

Confirmed diff between the two hosts (everything else identical — Docker 27.5.1, Sysbox 0.7.0, overlay2, kernel 6.17.0-101x-gcp, subuid/subgid):

	wei-tapp (fresh, broken)	wei-tapp2 (working)
br_netfilter loaded	no	yes
/proc/sys/net/bridge/bridge-nf-call-iptables	missing	present

Fix (applied on wei-tapp)

echo br_netfilter > /etc/modules-load.d/br_netfilter.conf
modprobe br_netfilter

After loading + restarting the runner: runner healthy, runner-bridge created with icc=false, cross-sandbox isolation active.

Action item — 0g-tapp CVM build

This belongs in the CVM provisioning (0g-tapp gcp-cvm/build-gcp-tapp.sh, next to ENABLE_SYSBOX): persist br_netfilter via /etc/modules-load.d/ so every new CVM has it at boot. Without it, every fresh CVM running this compose will runner-crash-loop. wei-tapp2 only worked because the module happened to be loaded already.

(Found while redeploying this compose to wei-tapp. This PR is correct; the gap is host-side, but it's this PR that makes the dependency load-bearing — worth documenting in the deploy runbook / CLAUDE.md host prerequisites too.)