Codex auth is structurally broken: single-use refresh token stored as a snapshot secret, no successful run since 2026-07-03 #64

Summary

The codex agent has not completed a run since 2026-07-03. Every invocation since
2026-07-06 has failed on OAuth refresh, four for four. The agent-fallback chain hides
this: the run is recorded under whichever agent eventually answered, so the dashboard shows
a healthy-looking outcome and nothing surfaces the fact that the configured primary agent
is dead.

Evidence

runs rows attributed to codex:

Every codex invocation after that appears only inside another run's fallback_attempts:

Stderr from the 08-28 run:

ERROR codex_login::auth::manager: Failed to refresh token: Your access token could not be
refreshed because your refresh token was already used. Please log out and sign in again.

The reason recorded in fallback_attempts is the generic turn.failed — the 401 is only
visible in the container log, not in the DB, so this was invisible to the dashboard for
eight weeks.

Root cause

Credentials are supplied as CODEX_AUTH_JSON_BASE64 — a base64 snapshot of auth.json,
stored as a repo secret in labro-rssrn and rendered into deploy/.env by
upgrade-image.yml / labro-restart.yml.

OAuth refresh tokens are single-use. The first refresh inside the container consumes the
stored token and issues a new one, which lives only in the container's ephemeral
~/.codex/auth.json. The next container recreate restores the original snapshot — a token
that has already been spent — and the refresh 401s. From then on there is no path back:
the stored secret can never again be valid.

So this is not "the secret went stale and needs rotating". Re-authenticating and re-storing
auth.json buys exactly one refresh cycle before the same failure recurs. The mechanism is
structurally incompatible with a snapshot-in-a-secret.

Options

1. Persist auth.json outside the container. Mount it from /opt/labro/data/ so the
   refreshed token survives recreates, and seed the file once by hand. Cheapest fix; keeps
   the ChatGPT-subscription auth path. Needs the file to be writable by the container and
   excluded from any rsync that would clobber it.
2. Switch codex to an API key (CODEX_API_KEY, already a declared secret). API keys do
   not rotate, so the snapshot problem disappears entirely — at the cost of usage-based
   billing rather than the subscription.
3. Drop codex from the model lists. It is the primary for all three *-proactive
   projects and nothing else. If neither of the above is worth the effort, removing it makes
   the configuration honest and saves the wasted first attempt on every proactive run.

Related

Whatever is chosen, the masking is a separate problem: a configured agent can fail on every
single invocation for two months without anything reporting it. That belongs with #60, which
already covers skipped hiding crashes and total source failures — same class of defect, in
the agent layer rather than the task-source layer. Worth cross-referencing rather than
solving here.