docs(oseps): OSEP-0021 Multi-Sandbox Egress Control Plane - #1592
docs(oseps): OSEP-0021 Multi-Sandbox Egress Control Plane#1592Pangjiping wants to merge 8 commits into
Conversation
Subject abstraction for a single egress control plane serving N sandboxes sharing one host/network domain. pod profile observes fastlet's network state store for identity, receives policies over fast-sandbox's proxy route, and loads credentials from Secret volumes; deny-first lifecycle keeps create-then-configure fail-closed. Zero fast-sandbox API impact.
github-actions
Bot
added
documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 748d694960
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| SRV->>C: CreateSandbox (no policy fields) | ||
| F->>S: slot bound (slot.json written — existing logic) | ||
| Note over E: fsnotify: bound slot | ||
| E->>E: Register subject: deny-first rules, resolv.conf → gateway |
There was a problem hiding this comment.
In the creation sequence, the sandbox's slot becomes bound before the fsnotify handler installs its deny rules. Because the proposal also leaves Fastlet admission and lifecycle phases untouched, traffic can traverse the Pod network during the interval between lines 134 and 136, so the claimed create-then-configure fail-closed guarantee does not hold. Install a permanent default-deny rule for unregistered sandbox sources before slots can be bound, or gate network activation on egress registration.
AGENTS.md reference: AGENTS.md:L117-L120
Useful? React with 👍 / 👎.
| SRV->>SEC: create Secret (credentials) | ||
| SEC->>E: kubelet sync → load to memory |
There was a problem hiding this comment.
Fastlet Pods are pre-created, but creating a per-sandbox Secret after slot allocation does not add that Secret to an already-running Pod's immutable volume sources. The proposal neither predeclares the Secret nor defines an aggregate Secret that is created with the Pod and subsequently updated, so SEC->>E cannot occur for dynamically created sandboxes and credential-proxy initialization will stall. Specify a mountable pre-provisioned source or another dynamic delivery mechanism.
AGENTS.md reference: AGENTS.md:L44-L46
Useful? React with 👍 / 👎.
| else Credential update | ||
| U->>SEC: update Secret | ||
| SEC->>E: kubelet sync → inotify |
There was a problem hiding this comment.
When a user invokes the existing /credential-vault POST/PATCH API, the SDK only has the resolved egress endpoint and route headers; it has neither Kubernetes credentials nor a Secret identity to perform U->>SEC. This also contradicts the earlier control-path table, which says the proxy route carries runtime vault operations. As written, existing SDK credential updates cannot follow this sequence, so either the proxy endpoint must continue accepting and persisting those mutations or the public API and SDKs must change together.
AGENTS.md reference: AGENTS.md:L42-L47
Useful? React with 👍 / 👎.
| - **Update** on policy push (proxy route): DNS policy swap + one atomic nft batch (delete+add in a single `nft -f` transaction). Credential updates via Secret volume refresh (inotify), memory-only. | ||
| - **Unload** on slot-file deletion (fastlet's existing release path): detach → deny → free. | ||
| - **Race handling**: a push for an unknown UID is cached as pending (with TTL) until the slot appears; both sides idempotent. Fencing mismatch (same UID, new generation) discards old state — a reset can never carry old policy into a new sandbox. | ||
| - **Recovery**: egress restart → rescan slot store, every live subject re-enters `denying`; server reconciliation re-pushes policies. No platform replay, no fastlet involvement. |
There was a problem hiding this comment.
The recovery path requires the server to re-push each live subject's policy, but the design sends no policy to FastPath, introduces no cluster-side carrier, and specifies no durable server store. If the server restarts before or with egress, it can enumerate sandboxes but cannot reconstruct their policies, leaving every recovered subject permanently in denying. Define where the policy is durably stored and how reconciliation retrieves it.
AGENTS.md reference: AGENTS.md:L117-L121
Useful? React with 👍 / 👎.
| - **Register** on observing a bound slot (identity + fencing from `Owner`); **denying** state installs deny-first rules immediately (nft sets empty, resolv.conf → gateway, REDIRECT + forward rules) — the subject is fully blocked until policy lands. | ||
| - **Update** on policy push (proxy route): DNS policy swap + one atomic nft batch (delete+add in a single `nft -f` transaction). Credential updates via Secret volume refresh (inotify), memory-only. | ||
| - **Unload** on slot-file deletion (fastlet's existing release path): detach → deny → free. | ||
| - **Race handling**: a push for an unknown UID is cached as pending (with TTL) until the slot appears; both sides idempotent. Fencing mismatch (same UID, new generation) discards old state — a reset can never carry old policy into a new sandbox. |
There was a problem hiding this comment.
Policy requests carry only X-Fast-Sandbox-Uid, while fencing information arrives independently from the slot store. Therefore, when the same UID is rebound with a new generation, egress cannot distinguish a delayed or pending old-generation push from a policy intended for the new subject, so it cannot implement the stated stale-policy discard guarantee. Include the generation in an authenticated proxy-supplied field and key pending updates by UID plus generation.
AGENTS.md reference: AGENTS.md:L117-L120
Useful? React with 👍 / 👎.
| E->>E: Register subject: deny-first rules, resolv.conf → gateway | ||
| SRV->>C: ResolveEndpoint(component="egress") | ||
| C-->>SRV: route + route credential | ||
| SRV->>P: PUT /v2/sandboxes/{uid}/components/egress/policy |
There was a problem hiding this comment.
The server initialization sequence depends on PUT /policy, but specs/egress-api.yaml defines only GET, PATCH, and DELETE for /policy. Since PATCH accepts only a rule array and cannot transmit the complete creation policy including defaultAction, the documented contract provides no equivalent operation. Relying on the implementation-only PUT while declaring the spec unchanged creates contract drift; add the replace operation to the source-of-truth spec or define a separate internal initialization endpoint.
AGENTS.md reference: AGENTS.md:L42-L48
Useful? React with 👍 / 👎.
…s access Replace the v2 component-route prefix with a dedicated sandboxfleets route handled by fastlet-proxy (parseTarget branch, host upstream, UID header); independent of the component Components map.
…ow protection - Security boundary: policy operations stay usable while a subject is denying (management plane never traverses sandbox traffic paths) - Creation window: OpenSandbox runtime driver probes egress healthz inside EnsureSandbox before container creation; existing drivers untouched
… Secret volume
Consistent with OSEP-0012: complete vault revisions are pushed by the
server over /v1/sandboxfleets/{sandboxId}/egress/credential-vault and
held memory-only in egress. Drops the Kubernetes Secret/kubelet sync
dependency (and the Secret volume from the egress container config).
Verified against containerd driver: slot store, pre-provisioning order, no NET_ADMIN, resolv.conf mount, route credentials, FastletTemplate all already present. NET_RAW is not dropped (runc default) - UDP spoofing mitigated by iifname binding or a preferred additive NET_RAW drop in the runtime driver.
…uestion Egress consumes nine slot fields (identity, fencing, dispatch, netns, veth, gateway, cidr, resolv path). Current source is fastlet's internal file store - not a public contract. Records two stabilization options (file contract vs kubelet-style read-only endpoint on fastlet's HTTP RPC server) as an open question for fast-sandbox.
|
关联：fast-sandbox#10（slot 数据稳定化契约归属的开放问题，待 fast-sandbox 表态） |
…mantics - TOC: Three -> Two Control Paths - Security boundary: NET_RAW drop belongs to the new OpenSandbox driver (existing sandboxes keep runc defaults); iifname is defense in depth - Sequence: add controller->fastlet create call - Route parsing: credential target semantics must match ResolveEndpoint - Creation window: healthz guarantees egress liveness; a fully deterministic guarantee needs subject-registered confirmation (recorded) - bwrap endpoint: mark TBD instead of n/a
Semicolon in a message label breaks mermaid parsing; replaced with a comma.
1 participant
OSEP-0021: Multi-Sandbox Egress Control Plane
Related RFC: #1582
Summary
Introduces a Subject abstraction so a single egress control plane can serve N sandboxes sharing one host/network domain (fast-sandbox Fastlet Pod, bwrap sessions). The existing single-sandbox sidecar profile is unchanged;
podprofile observes fastlet's existing network state store for identity, receives policies over fast-sandbox's own proxy-route mechanism, and loads credentials from Secret volumes (memory-only). A subject is deny-first from observation until its policy lands, so create-then-configure can never be fail-open.Key design points
denying->activestate machine, atomic policy swaps, fencing-based reset protectionReviewer focus