docs: add a blue-green cutover runbook and the re-derivation checklist#143
docs: add a blue-green cutover runbook and the re-derivation checklist#143LeonmanRolls wants to merge 2 commits into
Conversation
The containers are stateless and all indexed state lives in PostgreSQL, so a release whose indexing logic changed rewrites the database that is currently answering queries, and every intermediate state is visible to readers. This runbook describes filling an isolated second database, gating on it, and cutting the API over. Most releases do not need it, so the runbook opens with the decision. A change is additive when its backfill only inserts rows no existing answer depends on: a new contract or source family with its own start block, whose events do not feed the derived state of an entity that is already indexed. A change is re-derivation, and needs isolation, when it can rewrite what existing queries already return: an earlier manifest start block, a new event on an already-watched contract, a wider runtime watch scope, adapter changes for an adapter not classified `stateless_raw_fact`, a migration that rewrites rows, or a projection rebuild behind a consumer-facing route. The adapter classification is the discriminator rather than the apparent size of the diff. Two preconditions are hard gates. An image that records the code of every watched contract on every block writes rows proportional to watched contracts times blocks, which exhausts a volume rather than filling it. And an image whose runtime watch scope is narrower than the manifests and discovery edges declare fills the new database quietly with the wrong corpus, while passing every check that compares one pipeline stage to the stage before it. The cutover gates on `inspect data-completeness`; `/v1/status` cannot substitute, because it reports the stored canonical checkpoint whenever the projection queue is empty. The old database's indexer and worker keep running through a bake window, so rolling back means repointing at a database that is still current rather than one that has been falling behind since the cutover.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 660e56fa4d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Rollback is therefore the cutover in reverse: repoint `BIGNAME_DATABASE_URL` at blue | ||
| and restart the API. |
There was a problem hiding this comment.
When the promoted release includes API/storage migrations or new query expectations, this rollback recipe only changes BIGNAME_DATABASE_URL, leaving the new API binary running against blue, which this runbook never migrated. The API opens storage and warms SQL paths at startup, so pointing the new artifact at the old schema/projection shape can fail the rollback or keep serving new-code semantics instead of restoring the old release; include restoring the previous API image/config, or explicitly limit this shortcut to data-only rollbacks.
Useful? React with 👍 / 👎.
| - **Adapter logic changes for an adapter that is not classified | ||
| `stateless_raw_fact`.** Adapters classified `stateful_closure_required` or |
There was a problem hiding this comment.
This exception makes the replay classification the discriminator for adapter logic changes, but a stateless_raw_fact adapter can still rewrite normalized events for already-indexed raw facts when its decoding or normalization changes. For example, a stateless reverse-claim change can alter existing primary-name answers; replaying that in place exposes a partial old/new derived state to readers, so classification should only bound the replay closure, not exempt stateless adapter changes that affect existing routes.
Useful? React with 👍 / 👎.
| has actually been observed, that no pipeline stage is stalled or failing, and that | ||
| the projections are non-empty. |
There was a problem hiding this comment.
In the implementation and the data-completeness runbook, projections_non_empty only checks that normalized_events and name_current are non-empty; it does not prove that every projection table backing consumer routes, such as primary names, resolver, permissions, or record inventory, has content. For cutovers that promote routes backed by those projections, this sentence overstates what the gate proves and can make an empty route-specific projection look covered by the database gate.
Useful? React with 👍 / 👎.
|
|
||
| ## Cutting over | ||
|
|
||
| Only the API moves. Repoint its `BIGNAME_DATABASE_URL` at green and restart it. |
There was a problem hiding this comment.
For releases that change API handlers, generated queries, or schema expectations, this cutover step restarts whichever API artifact is currently serving blue and only changes its database URL. Green was migrated and filled by the promoted indexer/worker, so leaving the old API binary in place can either fail on the new schema or keep serving old route semantics from the new corpus; the cutover needs to deploy the promoted API image/config as part of moving traffic.
Useful? React with 👍 / 👎.
| the runtime watch scope is narrower than the manifests and discovery edges | ||
| declare, green will fill quietly and completely with the wrong corpus, and it | ||
| will pass every check that compares one pipeline stage to the stage before it. | ||
| Confirm with `bigname-worker inspect watch-plan --json`. |
There was a problem hiding this comment.
inspect watch-plan reads the persisted manifest/discovery watch plan, but it does not inspect the indexer's selected runtime scope or BIGNAME_INDEXER_HASH_PINNED_BACKFILL_ADAPTER_SYNC; in auto/raw-only modes the indexer can run with manifest_declared_only while this command still shows discovery-edge targets from storage. In that environment this precondition can look green even though the live indexer will not watch the corpus the runbook is trying to prove, so the runbook should require checking the indexer run mode/logged runtime_watch_scope or the final coverage gate instead.
Useful? React with 👍 / 👎.
…in the blue-green runbook Five review corrections to the blue-green cutover runbook. Cutover and rollback were described as connection swaps only. apps/api is a versioned in-repo artifact, so a release that changes API handlers, queries, or schema expectations must move the API image or config too — a URL swap alone leaves the old binary reading the other database's differently shaped data. Both sections now split the connection-swap shortcut (data-only releases) from the deploy-the-artifact path (API changed), and point at git diff over apps/api to tell which case applies. The re-derivation checklist exempted adapters classified stateless_raw_fact. That is wrong: classification bounds the replay closure size, not whether outputs change. EnsV1ReverseClaim is classified stateless_raw_fact yet produces primary-name state, so a decode change to it rewrites existing primary names when replayed in place. The bullet now keys re-derivation on whether the change alters what already-indexed raw facts derive to, with classification describing only how large the replay is. The watch-scope precondition told operators to confirm coverage with inspect watch-plan. That command reads the persisted manifest and discovery plan from storage, not the indexer's runtime scope, so an older image running manifest_declared_only passes it while watching far less than storage lists — the July incident's shape. It now directs operators to the indexer run mode and boot-log watch-scope fields, or to the coverage check in the cutover gate, and notes the image-version dependence. The gate summary claimed it proves the projections are non-empty. The gate checks exactly normalized_events and name_current, not the route-backing projections (primary names, resolver, permissions, record inventory). Reworded to say so, and aligned with the invalidation-queue and dead-letter checks the gate now carries.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: baa9c9a43e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| If the release changed API handlers, queries, or schema expectations — anything under | ||
| `apps/api` — the running API binary must move too. The API is a versioned in-repo | ||
| artifact, so a URL swap alone leaves the old binary reading green's new-shaped data. | ||
| Deploy the promoted API image or config pointed at green as the cutover step, not just | ||
| the connection string. `git diff --stat <blue-release>..<green-release> -- apps/api` | ||
| tells you which case you are in. |
There was a problem hiding this comment.
Fresh evidence in this revision is the new git diff ... -- apps/api discriminator. The API binary links bigname-storage and route handlers call storage query functions, so a release that changes migrations, schema expectations, or query semantics in crates/storage but not apps/api is classified here as a connection-only cutover; the old API artifact can then read green with old SQL/semantics or fail. Please make the decision based on the promoted API artifact and its dependencies, not only the apps/api path.
Useful? React with 👍 / 👎.
| projection invalidation queue is drained with no dead letters, and that | ||
| `normalized_events` and `name_current` are non-empty. |
There was a problem hiding this comment.
inspect data-completeness only loads projection_apply_cursors and max(projection_normalized_event_changes.change_id); it never reads projection_invalidations or projection_invalidation_dead_letters. Once the derive cursor is caught up, pending/claimed invalidations or terminal dead letters can still leave route projections stale while this sentence tells operators the queue is drained with no dead letters, so either narrow this proof or add a separate queue/dead-letter gate.
Useful? React with 👍 / 👎.
| Green is promotable when it is data-complete: | ||
|
|
||
| ```sh | ||
| bigname-worker inspect data-completeness \ | ||
| --database-url "$GREEN_DATABASE_URL" --json --fail-on-incomplete |
There was a problem hiding this comment.
This gate has no external head input: load_chain_completeness compares chain_lineage to chain_checkpoints stored in the same database. If green stops tailing after warm-up, loses RPC, or the indexer stalls after reaching an old checkpoint, the command can pass against that stale stored checkpoint, so this line can promote a database that is not current at the moment of cutover. Add an explicit provider/head freshness check or state that data-completeness is only a necessary gate.
Useful? React with 👍 / 👎.
| The gate is database-level. It proves that the database reconciled the chain | ||
| contiguously to head, that every contract the manifests and discovery edges declare |
There was a problem hiding this comment.
For a fresh green database where bootstrap/backfill skipped or failed older ranges, this gate can still pass after the live tailer observes current head: reconciliation_lineage_contiguous only checks the stored lineage span from its own floor to head, and code-observation coverage only requires at least one non-orphaned observation for each active target. That misses admitted start-block history, so cutover can silently serve a corpus missing old ownership, resolver, or reverse events; require completed bootstrap/backfill coverage from manifest starts or an explicit range audit before promotion.
Useful? React with 👍 / 👎.
|
|
||
| The gate is database-level. It proves that the database reconciled the chain | ||
| contiguously to head, that every contract the manifests and discovery edges declare | ||
| has actually been observed, that no pipeline stage is stalled or failing, that the |
There was a problem hiding this comment.
inspect data-completeness only evaluates normalized_replay_cursors rows that already exist: an active chain with raw logs but no raw_fact_normalized_events cursor produces no failed or lagging cursor and can still pass if global content counts are non-empty. In that misconfigured or partially bootstrapped green database, normalization for a chain may never have run while this text says no pipeline stage is stalled or failing; require the expected replay cursor per active watched chain/profile before treating the pipeline as healthy.
Useful? React with 👍 / 👎.
Server-pipeline adversarial review (head baa9c9a, stacked on #139) — REQUEST_CHANGESHeadline for the imminent deploy: this runbook is NOT safe to follow for the replay-version-9 cutover. docs/deployment.md (as amended by #169/#173) is the operative document. Three P1s:
Selected P2s: the cutover step ("repoint the API's BIGNAME_DATABASE_URL at green, keep blue's indexer/worker on blue") is not executable under docker-compose.server.yml's single shared env anchor without a stack split it never describes; the "images with the live-scope widening always widen and log it" check misclassifies every current image (#141 open; raw-only never widens; auto widens only post-handoff; boot always logs manifest_declared_only); five codex P2s on the gate's semantics stand for re-verification against #139; a cross-PR contradiction with #142 on cold-start behavior (the truth is per-family); "promotable release" adds an unqualified third sense of the glossary-flagged "promotion" overload. Verified sound and worth keeping: the additive/re-derivation classification concept, migrations-don't-run-on-boot + fail-closed healthcheck mechanics, the StatelessRawFact claim, and the fresh-green #173-gate analysis. Codex triage NOT satisfied (latest-head 5 P2s + unresolved earlier threads). Hard sequencing: rebase/retarget after #139 resolves, then rewrite the v9-overlap sections to defer to deployment.md. |
3 participants
Stacked on #139 — it references
inspect data-completeness, which that PR introduces. Base retargets tomainonce #139 merges. The only commit here is the runbook.Why
The containers are stateless; all indexed state lives in PostgreSQL. So a release whose indexing logic changed does not merely start running new code — it begins rewriting the database that is currently answering queries, and every intermediate state is visible to readers. There is no isolation between "deploying" and "mutating production data".
docs/runbooks/blue-green-cutover.mddescribes filling an isolated second database with the new release, gating on it, and moving the API across.The decision comes first
Most releases do not need a second database, and standing one up when it isn't needed is expensive. So the runbook opens with the classification rather than the procedure.
Additive — the backfill only inserts rows that no existing answer depends on. The clear case is a new contract or source family with its own admitted start block, whose events do not feed the derived state of an entity that is already indexed. Deploy in place.
Re-derivation — the change can rewrite what existing queries already return, so served answers regress while the work is in flight. Requires isolation. The triggers:
stateless_raw_fact, sincestateful_closure_requiredandcontextual_dependency_requiredadapters replay through a closure over prior stateThe adapter classification is the discriminator. Judging by how large the diff looks is how a re-derivation gets deployed in place.
The runbook also records why a widening watch scope specifically favours a fresh database: backfill jobs select targets from the set active at job creation, so a contract discovered after a job began was never in that job's scope. A database backfilled once with the corrected scope has no such ordering problem.
Two hard preconditions
Both produce a green database that is useless, in different ways:
watched contracts x blocksrather than to chain activity. Over a large watch set and a long history that exhausts the volume rather than filling it.Precondition 3 is softer but easy to get wrong: size the new database against the corpus you intend to have, not against the old one. If the release exists to index contracts the old database never watched, the new one will be larger.
Cutover and rollback
The gate is
inspect data-completeness --fail-on-incomplete./v1/statuscannot substitute, since it reports the stored canonical checkpoint whenever the projection queue is empty, making an empty database indistinguishable from a complete one.Only the API moves at cutover. The old database's indexer and worker keep running through a bake window — that is the property that makes rollback safe. Roll back and you are repointing at a database that is still current. Stop them at cutover and every minute of the bake window makes rollback worse, because the only database you can return to is falling further behind head.
Scope
Host-agnostic: expressed in
BIGNAME_DATABASE_URL,bigname-worker migrate, the three services, and the gate. It states plainly that a passing gate is a necessary condition for a cutover, not a sufficient one — it does not exercise routes or compare answers between the two databases.