feat: spill Codex traffic to a fallback API key at a usage ceiling

docs/GROK_MODELS.md

@@ -81,6 +81,10 @@ If the resolved secrets contain no legacy xAI refresh token, sandbox preparation
 The sandbox never receives the refresh token. Broker responses use `Cache-Control: no-store`, and
 the endpoint rejects user and service credentials in favor of the matching session's sandbox token.
 
+A session that can see an `XAI_API_KEY` secret skips this path entirely: no marker, no sentinel, no
+plugin. Grok calls then go straight to xAI's metered API with that key. Delete the secret to return
+to the SuperGrok subscription.
+
 ---
 
 ## Deployment and Rollout

docs/HOW_IT_WORKS.md

@@ -661,7 +661,9 @@ scoped OAuth. New sessions use an explicit choice, then a provider-account defau
 retain legacy scoped OAuth or API-key behavior. Setting a default affects only future sessions;
 operators may remove legacy keys after legacy-bound sessions are no longer needed. See
 [Using OpenAI Models](./OPENAI_MODELS.md) and
-[Using Grok with a SuperGrok Subscription](./GROK_MODELS.md).
+[Using Grok with a SuperGrok Subscription](./GROK_MODELS.md). A plain provider API key visible to
+the session (`OPENAI_API_KEY`, `XAI_API_KEY`) disables broker mode for that provider and is injected
+into the sandbox like any other secret, because OpenCode reads those variables directly.
 
 > **LLM API keys** (e.g., `ANTHROPIC_API_KEY` for Claude models) are added as global secrets. A
 > deployment can instead configure `anthropic_api_key` in Terraform to inject one fleet-wide key

docs/OPENAI_MODELS.md

@@ -56,6 +56,58 @@ choice for future runs.
 
 ---
 
+## Spilling over before the subscription runs out
+
+A ChatGPT subscription that hits its Codex quota fails the session outright:
+`Execution failed: The usage limit has been reached...`. Two optional secrets let a deployment keep
+working on a per-token key, and cap how much of the subscription sandboxes may take in the first
+place:
+
+| Secret Name                       | Value                                                                     |
+| --------------------------------- | ------------------------------------------------------------------------- |
+| `OPENAI_API_KEY_FALLBACK`         | A platform API key, used only as a spillover                              |
+| `OPENAI_SUBSCRIPTION_MAX_PERCENT` | Optional share of a rate-limit window sandboxes may consume (default 100) |
+
+`OPENAI_API_KEY_FALLBACK` is deliberately a separate name from `OPENAI_API_KEY`: the latter selects
+metered billing for the whole session and is stripped from sessions routed to a subscription, while
+this one rides along unused until the subscription cannot answer. Set
+`OPENAI_SUBSCRIPTION_MAX_PERCENT` to `80` to reserve the last fifth of each window for whoever else
+uses that ChatGPT account.
+
+A sandbox sends OpenAI traffic to the subscription until one of these happens, then latches to a
+successful fallback path for the rest of its life:
+
+- when `OPENAI_SUBSCRIPTION_MAX_PERCENT` is below `100`, usage is already at or above that ceiling
+  before the first turn. The percentage comes from `GET /backend-api/wham/usage`, which reports both
+  windows without consuming either
+- a Codex response reports either window at or above the ceiling. On a successful response the
+  in-flight reply is kept and only the next request moves over, because a started stream cannot be
+  replayed
+- Codex answers `429` with a quota signal: a recognized primary/secondary
+  `x-codex-rate-limit-reached-type`, a specific usage-limit-reached message, or a window at or above
+  the ceiling. That request is retried on the fallback key immediately
+- the control plane reports that the subscription credential is unusable or requires reconnection.
+  Sandbox-auth, transient broker, network, timeout, configuration, and storage failures do not spend
+  the fallback key
+
+A plain `429` with no quota signal is passed through untouched, so short-window throttling does not
+spend money. Codex tracks a short (roughly 5-hour) and a weekly window, and the higher usage of the
+two decides. An unparseable ceiling is ignored with a log line and treated as 100. If the usage
+probe fails, the sandbox stays on the subscription and relies on response headers instead.
+
+`gpt-5.3-codex-spark` is subscription-only, so its platform fallback uses `gpt-5.3-codex`. Other
+allowed models are sent unchanged. If the platform rejects a fallback request, the latch is cleared
+and the next turn retries the subscription instead of remaining on a permanently failing paid path.
+
+Every switch is logged in the sandbox logs as
+`[codex-auth-plugin] spilling OpenAI traffic over to OPENAI_API_KEY_FALLBACK: <reason>`.
+
+One caveat: after a successful spillover, the latch lasts as long as the sandbox even if the
+subscription window resets under it. OpenCode also reports OpenAI token costs as `0` because the
+Codex proxy zeroes them at startup.
+
+---
+
 ## How It Works
 
 The OpenAI device authorization result is encrypted with `PROVIDER_ACCOUNTS_ENCRYPTION_KEY` in the
@@ -93,6 +145,13 @@ Open-Inspect.
 Confirm that the selected/default OpenAI account is active and the account is verified. If the
 session explicitly uses API-key mode, confirm `OPENAI_API_KEY` is available in its secret scope.
 
+### "The usage limit has been reached"
+
+The ChatGPT subscription hit its Codex quota. Wait for the window to reset, switch the session to
+another provider's model, or configure a spillover key
+([Spilling over](#spilling-over-before-the-subscription-runs-out)) so sessions continue on metered
+billing.
+
 ### "Token refresh failed" errors
 
 The OAuth grant may have been revoked, expired, or rotated elsewhere. Use **Reconnect** on the

docs/SECRETS.md

@@ -198,16 +198,18 @@ from it, even after you rotate the secret. Two guidelines:
 
 ## Common Examples
 
-| Key                 | Scope  | Purpose                                               |
-| ------------------- | ------ | ----------------------------------------------------- |
-| `ANTHROPIC_API_KEY` | Global | Claude API access                                     |
-| `OPENAI_API_KEY`    | Global | OpenAI API access when a session selects API-key mode |
-| `XAI_API_KEY`       | Global | xAI API access when a session selects API-key mode    |
-| `DEEPSEEK_API_KEY`  | Global | DeepSeek API access                                   |
-| `ZHIPU_API_KEY`     | Global | Z.AI Coding Plan GLM access                           |
-| `DATABASE_URL`      | Repo   | Database connection string                            |
-| `AWS_ACCESS_KEY_ID` | Repo   | AWS credentials for a specific project                |
-| `STRIPE_SECRET_KEY` | Repo   | Stripe API key for a specific project                 |
+| Key                               | Scope  | Purpose                                                                                                                                |
+| --------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------- |
+| `ANTHROPIC_API_KEY`               | Global | Claude API access                                                                                                                      |
+| `OPENAI_API_KEY`                  | Global | OpenAI API access when a session selects API-key mode                                                                                  |
+| `XAI_API_KEY`                     | Global | xAI API access when a session selects API-key mode                                                                                     |
+| `DEEPSEEK_API_KEY`                | Global | DeepSeek API access                                                                                                                    |
+| `ZHIPU_API_KEY`                   | Global | Z.AI Coding Plan GLM access                                                                                                            |
+| `OPENAI_API_KEY_FALLBACK`         | Any    | Spillover once the ChatGPT subscription reaches its ceiling ([guide](OPENAI_MODELS.md#spilling-over-before-the-subscription-runs-out)) |
+| `OPENAI_SUBSCRIPTION_MAX_PERCENT` | Any    | Share of a Codex rate-limit window sandboxes may consume (default 100)                                                                 |
+| `DATABASE_URL`                    | Repo   | Database connection string                                                                                                             |
+| `AWS_ACCESS_KEY_ID`               | Repo   | AWS credentials for a specific project                                                                                                 |
+| `STRIPE_SECRET_KEY`               | Repo   | Stripe API key for a specific project                                                                                                  |
 
 ---
 

packages/control-plane/src/routes/model-provider-accounts.test.ts

@@ -0,0 +1,23 @@
+import { describe, expect, it } from "vitest";
+import type { ModelProviderAccountBrokerErrorCode } from "../auth/model-provider-account-broker";
+import { modelProviderBrokerHttpStatus } from "./model-provider-accounts";
+
+describe("modelProviderBrokerHttpStatus", () => {
+  it.each([
+    ["account_not_found", 404],
+    ["upstream_retry_safe", 502],
+    ["provider_unavailable", 503],
+    ["exchange_busy", 503],
+    ["account_inactive", 409],
+    ["account_archived", 409],
+    ["provider_mismatch", 409],
+    ["credential_not_found", 409],
+    ["credential_invalid", 409],
+    ["reconnect_required", 409],
+  ] satisfies Array<[ModelProviderAccountBrokerErrorCode, 404 | 409 | 502 | 503]>)(
+    "maps %s to HTTP %i",
+    (code, status) => {
+      expect(modelProviderBrokerHttpStatus(code)).toBe(status);
+    }
+  );
+});

packages/control-plane/src/routes/model-provider-accounts.ts

@@ -19,6 +19,7 @@ import { modelProviderAccountAdapterRegistry } from "../auth/model-provider-acco
 import {
   ModelProviderAccountBroker,
   ModelProviderAccountBrokerError,
+  type ModelProviderAccountBrokerErrorCode,
 } from "../auth/model-provider-account-broker";
 import { ModelProviderAccountStore } from "../db/model-provider-accounts";
 import { D1ModelProviderAccountAtomicWriter } from "../db/model-provider-account-atomic-writer";
@@ -440,6 +441,22 @@ async function handleLegacyProviderAccess(
   });
 }
 
+export function modelProviderBrokerHttpStatus(
+  code: ModelProviderAccountBrokerErrorCode
+): 404 | 409 | 502 | 503 {
+  switch (code) {
+    case "account_not_found":
+      return 404;
+    case "upstream_retry_safe":
+      return 502;
+    case "provider_unavailable":
+    case "exchange_busy":
+      return 503;
+    default:
+      return 409;
+  }
+}
+
 async function handleProviderAccess(
   _request: Request,
   env: Env,
@@ -489,9 +506,7 @@ async function handleProviderAccess(
     return json(await broker.getAccess(binding.providerAccountId, parsedProvider));
   } catch (cause) {
     if (cause instanceof ModelProviderAccountBrokerError) {
-      const status =
-        cause.code === "account_not_found" ? 404 : cause.code === "upstream_retry_safe" ? 502 : 409;
-      return error(cause.message, status);
+      return error(cause.message, modelProviderBrokerHttpStatus(cause.code));
     }
     return error("Provider access unavailable", 503);
   }

packages/sandbox-runtime/src/sandbox_runtime/opencode_server.py

@@ -319,15 +319,19 @@ def _install_skills(self, workdir: Path) -> set[str]:
         return installed
 
     def _setup_managed_oauth(self) -> None:
-        """Write OpenCode OAuth sentinels for control-plane-managed providers."""
+        """Sync OpenCode auth entries with the control-plane-managed providers.
+
+        Writes an OAuth sentinel for every provider the control plane brokers,
+        and drops sentinels for providers it no longer brokers. The removal
+        matters on snapshot restores: OpenCode prefers an OAuth entry over a
+        provider's API key env var, so a leftover sentinel would shadow a key
+        the operator installed to replace the subscription.
+        """
         openai_managed = os.environ.get("OPENAI_OAUTH_MANAGED")
         xai_managed = os.environ.get("XAI_OAUTH_MANAGED")
-        if not openai_managed and not xai_managed:
-            return
 
         try:
             auth_dir = Path.home() / ".local" / "share" / "opencode"
-            auth_dir.mkdir(parents=True, exist_ok=True)
 
             oauth_entry = {
                 "type": "oauth",
@@ -344,15 +348,15 @@ def _setup_managed_oauth(self) -> None:
             auth_file = auth_dir / "auth.json"
             tmp_file = auth_dir / ".auth.json.tmp"
 
-            existing_entries = {}
+            existing_entries: dict[str, Any] = {}
             if auth_file.exists():
                 try:
                     existing = json.loads(auth_file.read_text())
                     if isinstance(existing, dict):
                         existing_entries = existing
                 except (OSError, json.JSONDecodeError):
                     self.log.warn("managed_oauth.existing_auth_invalid")
-            existing_entries = {
+            retained = {
                 key: value
                 for key, value in existing_entries.items()
                 if not (
@@ -361,7 +365,11 @@ def _setup_managed_oauth(self) -> None:
                     and key not in entries
                 )
             }
-            entries = {**existing_entries, **entries}
+            merged = {**retained, **entries}
+            if not entries and merged == existing_entries:
+                return
+            entries = merged
+            auth_dir.mkdir(parents=True, exist_ok=True)
 
             # Write to a temp file created with 0o600 from the start, then
             # atomically rename so the target is never world-readable.