feat: add a read-only data-completeness gate for promotion and cutover by LeonmanRolls · Pull Request #139 · ensdomains/bigname

LeonmanRolls · 2026-07-10T04:09:07Z

Why

On 2026-07-06 the Sepolia deployment served zero names for roughly two weeks on a stale, crash-looping build. /v1/status reported projection_lag_blocks: 0 throughout.

That is not a bug in the endpoint so much as a category error in using it as a readiness signal. latest_projected_block is set to the stored canonical checkpoint whenever the projection queue is empty (crates/storage/src/identity_facade/status.rs), so an empty work queue reads as caught up to head. And every cursor in the pipeline measures itself against the previous stage's frontier — normalized_replay_cursors against the raw-log head, projection_apply_cursors against the change log — so when an upstream stage stalls, or the watch set silently excludes targets, all of them faithfully report "done".

An empty database and a complete database are indistinguishable to /v1/status. Nothing in the tree currently compares what was indexed against what the manifests and discovery edges say should have been indexed.

What

bigname-worker inspect data-completeness, alongside the existing inspect watch-plan and manifest-drift audit. Read-only: it opens the inspect connection with default_transaction_read_only = on and writes nothing.

bigname-worker inspect data-completeness --json                       # report, always exit 0
bigname-worker inspect data-completeness --json --fail-on-incomplete  # exit 1 when incomplete

Checks, in dependency order:

Check	Passes when
`reconciliation_frontier_at_head`	every active chain has a frontier row whose checkpoint hash/number resolves to an exact retained canonical/safe/finalized lineage anchor and whose head lag is within `±--max-head-lag-blocks` (default 8)
`reconciliation_lineage_contiguous`	distinct canonical/safe/finalized block numbers equal `head - floor + 1`, no retained canonical height has an additional non-orphaned hash, and every canonical row above the floor links by `parent_hash` to a canonical row at the preceding height
`reconciliation_history_from_declared_start`	each active watched chain's lineage floor is at or below the earliest finite start block its active targets declare; an all-open-ended chain fails
`stored_lineage_backfill_coverage`	every retained-lineage interval identified by a persisted bounded-backfill job has durable coverage facts for every active log-producing watched tuple, using the stored-lineage writer's anti-join and chunk boundaries, and every intersecting topic-filtered completed job still matches the active manifest ABI topic set
`watch_set_code_observation_coverage`	there is ≥1 active target, and every active target — manifest-payload roots, contracts, and proxy implementations unioned with the materialized watch view and active discovery edges — has ≥1 non-orphaned `raw_code_hashes` observation with an exact retained non-orphaned lineage anchor at or after its declared start block
`manifest_declared_targets_present`	every active manifest-payload root, contract, and proxy implementation has its matching materialized instance and open address whose start does not narrow the payload start; every proxy implementation also has the exact open, range-preserving managed `proxy_implementation` edge consumed by the watch view
`discovery_targets_present`	every target endpoint of an open non-migration discovery edge has an open, range-preserving `contract_instance_addresses` row on the edge's chain, and every open resolver edge from an active registry source retains its matching active resolver target manifest; bounded edges remain history
`active_event_lineage_retained`	every matching serving-canonical normalized event for an active event-producing manifest source resolves by exact chain, block hash, and block number to a canonical/safe/finalized `chain_lineage` anchor
`normalization_no_failure`	no replay cursor for an active chain under the deployment profile inferred from the active manifest corpus carries a `last_failure_reason`
`normalization_caught_up_to_raw_head`	the active manifest corpus resolves to one supported deployment profile; every active chain with retained canonical raw logs has that profile's `raw_fact_normalized_events` cursor, each applicable cursor has reached its target, and a post-replay backlog starts no later than the raw-fact target plus one
`projection_apply_drained`	the named apply cursor, when present, exactly equals the retained change-log high-water mark (`max(change_id)`, or zero for an empty log); a non-empty log also requires the cursor
`projection_invalidations_drained`	`projection_invalidations` is empty — every enqueued invalidation applied and was deleted
`projection_no_dead_letters`	`projection_invalidation_dead_letters` is empty
`projection_replay_complete`	every current projection has a marker at this worker's `CURRENT_PROJECTION_REPLAY_VERSION`; before the durable projection-apply cursor exists, each marker must also cover the global target automatic bootstrap would request now
`active_dataset_non_empty`	every active manifest source that declares normalized-event output has a serving-canonical event matching its exact manifest ID, version, chain, namespace, source family, and declared event kind; every such namespace has `name_current` rows
`normalized_events_chain_id_present`	no non-orphaned `normalized_events` row has a NULL `chain_id`
`deferred_projection_indexes_present`	the eight deferred `normalized_events` projection indexes all exist and are valid (`indisvalid`/`indisready`)

The check that carries the gate

Two checks carry the declared-input gate. watch_set_code_observation_coverage proves every active target was observed, while stored_lineage_backfill_coverage proves every retained interval produced by bounded-backfill authority still has the durable watched-tuple fetch evidence required by stored-lineage promotion. reconciliation_history_from_declared_start and active_dataset_non_empty also measure against the declared world, while active_event_lineage_retained checks serving-canonical active events against durable canonical lineage. The other checks are mostly relative invariants: each compares one stage to the stage before it, so they can stay green while the pipeline faithfully processes an incomplete input. That is exactly what happened in July.

It works because code observations are keyed on the watch set rather than on activity — a watched target acquires at least one observation from intake even if it never emits a log, via the live tailer's missing-baseline pass. Coverage counts an observation only when its exact block identity is still anchored by retained non-orphaned lineage.

The original fourteen-check implementation produced this read-only Sepolia diagnosis:

data_complete: false
  [PASS] reconciliation_frontier_at_head
  [PASS] reconciliation_lineage_contiguous
  [PASS] reconciliation_history_from_declared_start
  [FAIL] watch_set_code_observation_coverage
           active_watched_target_count : 355
           unobserved_target_count     : 352
  [PASS] manifest_declared_targets_present
  [PASS] normalization_no_failure
  [PASS] normalization_caught_up_to_raw_head
  [PASS] projection_apply_drained
  [PASS] projection_invalidations_drained
  [PASS] projection_no_dead_letters
  [PASS] projection_replay_complete
  [PASS] active_dataset_non_empty
  [PASS] normalized_events_chain_id_present
  [PASS] deferred_projection_indexes_present

advisories:
  foreign_chains: []
  backfill_lifecycle: [sepolia failed_job=22 failed_range=22 incomplete=274 expired_lease=45]

Thirteen of fourteen pass. The three observed targets are the manifest-declared registry, registrar, and root; the 352 unobserved are discovered resolvers and subregistries that the runtime watch scope never widened to include. Their logs were never indexed and their derived state is missing. The 22 failed backfills are surfaced as an advisory, not gated.

First review follow-up

The second commit closes six holes found in review — four where the gate would approve an incomplete database, two where it would block a healthy one:

Projection application was proven only as far as the derive scan. The apply cursor reaching max(change_id) means the scan finished, but the resulting invalidations move through a separate claim/apply queue: a successful apply deletes its projection_invalidations row, an exhausted one moves to projection_invalidation_dead_letters. Added projection_invalidations_drained and projection_no_dead_letters (both require an empty table).
The apply check anchored on projection_apply_cursors with a cross join, so an empty cursor table returned no rows and passed even with a non-empty change log. max(change_id) is now loaded independently and the check fails when the change log is non-empty but no apply cursor exists.
normalization_caught_up_to_raw_head compared against the non-orphaned raw-log head, which includes observed logs replay is not eligible to consume; a candidate with trailing observed logs stuck failing. It now compares against the canonical raw-log head, mirroring the replay bounds (raw log and its lineage block both canonical/safe/finalized). The non-orphaned head is retained as a reported-only value.
The same check iterated existing cursor rows only, so a chain with retained canonical raw logs but no raw_fact_normalized_events cursor passed vacuously. It now requires the cursor to exist for any chain with a canonical raw-log head.
Closure/dependency-replay chains latch the raw_fact_normalized_events cursor permanently below the live head, so comparing it to the head meant the gate could never pass there (e.g. base-mainnet; Sepolia has no closure adapters, which is why the first run passed). The latch is detected by a post_replay_live_adapter_backlog cursor row; such chains require the raw-fact and backlog cursors to reach their own targets. The live tail beyond the backlog target has no cursor and is documented as a gate limitation.
watch_set_code_observation_coverage passed vacuously at zero active targets, so a manifest-less restore cleared the load-bearing check. It now requires ≥1 active watched target.

Second review follow-up

The third commit closes five more holes from the re-review — three false-PASS, one false-FAIL, and one that widened the checks against the declared world:

Negative head lag passed. head_lag_blocks <= max accepted a negative lag — a canonical checkpoint sitting behind the retained lineage head, i.e. a stale checkpoint writer or mixed restore. The frontier check now requires 0 <= lag <= max.
A manifest-declared chain absent from storage passed by absence. known_chains was checkpoints ∪ lineage, so a chain the active watch set declares but that is missing from both tables produced no row and every per-chain check passed vacuously. The evaluator now synthesizes a failing frontier row (missing_from_storage: true) for every active watched chain with no storage row.
Duplicate canonical heights were invisible. COUNT(DISTINCT block_number) cannot see two non-orphaned canonical/safe/finalized hashes at one height. A per-chain duplicate-height count is now loaded, and reconciliation_lineage_contiguous fails when it is non-zero.
A truncated-history database passed everything. The contiguity span runs from MIN(retained lineage), so a live-tail-only restore is internally consistent — contiguous span, caught-up cursors, non-empty projections. The new reconciliation_history_from_declared_start check requires each active watched chain's lineage floor to sit at or below the earliest finite start block its active targets declare; targets with an open-ended start impose no floor.
projections_non_empty counted global tables. Leftover rows from another chain, profile, or namespace satisfied it while a newly active chain had zero. Renamed to active_dataset_non_empty and scoped: each active watched chain must have normalized_events (by chain_id), and each namespace those chains produce must have name_current (by namespace, the finest dimension a chain-less name projection has).

Note: the negative-lag fix is superseded by the third commit — see below.

Third review follow-up

A paired adversarial review of the second commit surfaced thirteen findings plus four design decisions, each implemented at its recommended default. The gate grows to thirteen checks, with two advisories.

Namespace expectation from declared authority. active_dataset_non_empty derived expected namespaces from observed events, so a chain declared to produce two namespaces passed with one wholly missing. The expected (chain, namespace) set now comes from active manifest_versions rows.
Chain authority. The active chain set is the watch view unioned with active manifest_versions directly, so a partial restore that lost contract_instance_addresses rows cannot delete a chain from its own expectations. Per-chain gating applies to active chains only; a foreign or retired chain with residual storage rows is an advisory.
Orphaned and NULL-chain content. Content counts exclude orphaned normalized events; a NULL chain_id row is excluded from the per-chain counts (it would otherwise abort the read) and surfaced by a new normalized_events_chain_id_present check.
Rewound cursors. The caught-up determination reads the cursor's next/target pair — the pipeline's completion authority — instead of the monotonic last_completed, which a reorg rewind leaves high while lowering next below target.
Symmetric lag tolerance. The wave-2 zero floor false-failed live databases, where reconcile commits canonical lineage before advancing the checkpoint (a routine negative lag). The tolerance is now ±max; a larger gap in either direction still fails.
Projection replay markers. New projection_replay_complete requires all current projections to have a current_projection_replay_status marker at the running worker's CURRENT_PROJECTION_REPLAY_VERSION, matching the worker's bootstrap handoff (name_current is published first, so requiring only it lets a mid-bootstrap candidate pass). The latest stored version remains diagnostic output, but complete markers from an older image do not satisfy the current worker; the markers are the authority, so no per-projection non-empty check is added.
Deferred projection indexes. New deferred_projection_indexes_present verifies the eight normalized_events indexes a fresh replay drops and rebuilds.
Open-ended starts (history). A chain whose active targets are all open-ended now fails, since no history floor can be established; finite-start targets are unchanged.
Backfill lifecycle (advisory). Per-profile failed/incomplete/expired-lease counts are reported, not gated — without coverage-fact reconciliation a failed range cannot be told from one superseded by a retry, and the data-level checks already gate the outcome.
Coinbase SQL sample mode. Documented in the runbook as a non-promotable state (sparse lineage false-fails contiguity) rather than special-cased in code.
DB-backed storage tests now cover the duplicate-height CTE, the manifest-declared chain missing from checkpoints/lineage, the orphaned and NULL chain_id exclusion, the rewound cursor, and the deferred indexes.

Fourth review follow-up

A further review round closes six more holes — four false-PASS, one false-FAIL, and one where a declared target could disappear entirely. The gate grows to fourteen checks.

Apply-cursor keyed to name. projection_apply_drained anchored on the projection_apply_cursors table being non-empty, so a restore that kept an unrelated or stale cursor row while dropping the real normalized_events_to_projection_invalidations cursor read as drained (the stale row masking the missing one, or standing in for it above max(change_id)). Presence and lag are now keyed to that cursor name.
Code observation within the active range. watch_set_code_observation_coverage collapsed raw_code_hashes to (chain, address), so a pre-admission observation of a reused address satisfied a target whose declared start is later. The observation now must fall at or after the target's active_from_block_number.
Disconnected canonical lineage. reconciliation_lineage_contiguous was height-only (count == head - floor + 1 plus the duplicate-height count), blind to a retained canonical branch that is complete by height but whose parent_hash chain is broken. A per-chain count of canonical rows above the floor whose parent links to no canonical predecessor is now loaded, and a non-zero value fails contiguity.
Deferred indexes must be valid. deferred_projection_indexes_present read pg_indexes, which lists an index left invalid by a failed CREATE INDEX CONCURRENTLY (pg_index.indisvalid false). The read now joins pg_index/pg_class and requires indisvalid and indisready, so an unusable replay-critical index is not counted present.
Manifest-declared target coverage authority. The watch view reads materialized declaration and address rows, so a partial restore that lost either could drop the target from the view entirely. Direct targets now come from active manifest_versions.manifest_payload, including proxy implementations, and manifest_declared_targets_present fails when the matching manifest_contract_instances/implementation instance or live address row is absent or mismatched.
Non-event manifests excluded from content. active_dataset_non_empty derived its expected (chain, namespace) set from every active manifest_versions row, so execution/transport manifests that declare a namespace without event sources (the checked-in basenames_execution and basenames_l1_compat on ethereum-mainnet) forced a complete mainnet database to fail on (ethereum-mainnet, basenames). The expected set is now limited to active manifests that declare normalized-event outputs.
DB-backed and unit tests cover each shape: the in-range and pre-admission observation, the disconnected-parent count, an invalid index left by a failed concurrent unique build, the manifest-declared instance with only a deactivated address row, and the event-producing vs execution manifest split.

Synthesis follow-up

Merge commit 09f60c9 selects the direct manifest authority model and closes the Wave 5 findings:

Coverage unions active manifest-payload roots, contracts, and proxy implementations with materialized watch rows and discovery edges. Shared addresses use the strictest finite start for code-observation coverage.
manifest_declared_targets_present separately requires each payload target's matching declaration/implementation instance and a live address row matching both the manifest chain and declared address.
History uses starts read directly from manifest payloads, so a later contract_instance_addresses.active_from_block_number cannot raise the declared history floor.
Content is keyed to the exact active manifest identity and declared event kinds. A manifest-version rollout without re-derivation therefore correctly fails until events exist under the newly active source_manifest_id; residual rows from the prior version do not satisfy it.
Projection replay markers must use the running worker's replay version and carry a completed_normalized_target_block covering the global target automatic bootstrap would request now: the greater of the raw-fact replay target and persisted chain-checkpoint frontier.

Final review follow-up

Commit d08b166 resolves the three confirmed findings from the final four-comment review and records the writer-parity decision for the fourth:

Replay completion is checked against CURRENT_PROJECTION_REPLAY_VERSION; the latest stored version is reported separately as diagnostic state. Complete older-version markers fail.
Direct target authority is parsed from the active manifest payload, so losing a manifest_contract_instances child cannot shrink coverage, history, or the named presence check.
Proxy implementation addresses are direct targets with the proxy declaration's start block and require their own implementation instance and live address row.
The required replay target remains global. This matches the automatic projection-replay writer, whose marker and target are global; scoping only the inspector to active chains could approve a marker below the target the writer requests. Foreign chains remain advisory for per-chain frontier, history, normalization, and coverage checks.

DB-backed regressions cover missing declaration children, missing proxy-implementation address rows, and the global writer watermark. The worker regression covers complete markers from the prior replay version.

Previous review follow-up

Commit 547dc2c resolves all six confirmed findings from the latest review round. The gate now has sixteen named checks.

Discovery endpoints survive missing address materialization. Active discovery-edge endpoints are loaded independently of the runtime watch view; discovery_targets_present fails when a target loses its live address row instead of letting coverage/history silently shrink.
Projection replay respects durable apply handoff. Current-version replay markers must cover the bootstrap target only before the normalized_events_to_projection_invalidations cursor exists. After handoff, drained change/apply/invalidation state is the advancing writer authority and replay marker targets intentionally stay fixed.
Proxy implementations require their managed edge. manifest_declared_targets_present now rejects a materialized proxy/implementation pair whose exact active manifest_declared_proxy discovery edge is missing.
Raw-log-backed derived content initially required retained facts. Commit 547dc2c introduced a raw-log-specific predicate. After checking the documented minimal-retention contract, the current follow-up below superseded it with exact retained event lineage so compacted raw_logs remain valid.
Inactive-chain cursor failures are advisory. Replay failures and lag outside the active chain set no longer block promotion and remain visible in advisories.ignored_replay_cursors.
Replay cursor coverage became profile-aware. Commit 547dc2c keyed cursor selection to the active manifest corpus. The current follow-up below corrected Sepolia from sepolia-dev to the writer's sepolia profile and made replay admission and inspection share one authority.

DB-backed regressions at that commit covered vanished discovery addresses, missing proxy edges, the initial raw-log predicate, and the initial active-profile inference. The current follow-up below supersedes the latter two while retaining the apply-handoff, inactive-cursor, stale-profile, unresolved-profile, and named-check worker regressions.

Current review follow-up

Commit 8313d65 accounts for all six comments in the current review round. Five findings were direct writer/inspector mismatches; the discovery-edge finding required a narrower current-authority fix because bounded dynamic edges are legitimate retained history.

Active manifest payload targets now require an address row with both deactivated_at IS NULL and active_to_block_number IS NULL.
Manifest-managed proxy implementation edges must likewise remain open; a bounded managed edge no longer satisfies the active payload.
Replay admission and inspection now share one active-manifest deployment-profile inference function. The checked-in manifests/sepolia corpus selects the writer's sepolia cursor profile.
The raw-log-specific check is replaced by active_event_lineage_retained. Every matching active normalized event, including synthetic boundary events, must retain its exact canonical/safe/finalized lineage anchor. This preserves the documented minimal-retention mode, where raw_logs are staging and may be compacted after normalized replay and downstream durability.
discovery_targets_present now mirrors current runtime authority: open edges require open target addresses. Finite active_to_block_number edges remain valid historical discovery rows and are excluded from the current presence requirement.
Code observations used for target coverage must join an exact retained non-orphaned lineage row by chain, block hash, and block number.

DB regressions cover bounded manifest addresses, bounded managed proxy edges, open-edge/closed-address discovery, historical bounded discovery edges, minimal-retention event lineage, unanchored code observations, and the Sepolia replay profile.

Latest review follow-up

Commit 92087f7 resolves all four additional findings from the latest review round and grows the gate to seventeen named checks.

Serving-canonical active content. Active source counts now admit only canonical/safe/finalized normalized events, matching the projection/apply readers. An observed event on canonical lineage no longer satisfies content.
Range-preserving discovery authority. An open discovery edge requires an open target address whose start is absent or no later than the edge start. If the edge start is unknown, only an address with an unknown start can preserve it; bounded edges remain historical rather than current authority.
Checkpoint branch anchors. reconciliation_frontier_at_head now requires the checkpoint's exact (canonical_block_hash, canonical_block_number) to resolve to retained canonical/safe/finalized lineage in addition to the symmetric numeric lag bound.
Durable bounded-backfill evidence. New stored_lineage_backfill_coverage reconciles every retained range identified by a persisted bounded-backfill job — including incomplete or failed crash residue — through the same watched-tuple anti-join and 131072-block chunking used by stored-lineage promotion. Facts produced by a later successful retry satisfy the range; ordinary provider-fetched live lineage with no persisted bounded-backfill job remains exempt.

DB-backed regressions cover observed active events, narrowed and unknown discovery starts, wrong/missing/orphaned checkpoint hashes, and missing durable coverage before and after checkpoint advancement. Pure evaluator coverage confirms that either a checkpoint anchor mismatch or an uncovered retained backfill tuple fails data_complete.

Final authority follow-up

Commit b8f128f resolves all seven findings from the newest review round without adding another named check:

Manifest-declared address rows must preserve the payload start. A later finite address start, including a finite materialization of an unknown payload start, now fails manifest_declared_targets_present.
Manifest-managed proxy implementation edges must likewise start no later than the payload implementation target (or remain open-ended when that target is open-ended).
projection_apply_drained now rejects a named apply cursor ahead of the retained change-log high-water mark, including a non-zero cursor over an empty log. The derive writer scans only IDs greater than the cursor, so exact equality is required.
Open resolver discovery edges from active registry manifests remain authority when the matching active resolver target manifest disappears. discovery_targets_present reports the missing target manifest separately from address materialization.
Completeness inspection and stored-lineage promotion now call the same shared topic-set drift guard before trusting bounded-backfill coverage facts. Persisted topic-filtered job sets must exactly match the active manifest ABI; a missing persisted set fails closed.
Canonicality checks now reject any additional non-orphaned hash at a retained canonical/safe/finalized height, matching stored-lineage promotion rather than considering only serving-canonical siblings.
Latched replay now loads each cursor's inclusive range start and requires a post-replay backlog to start no later than raw_fact_normalized_events.target_block_number + 1; an earlier overlap is safe, while a later start exposes an unnormalized gap.

DB-backed regressions cover narrowed manifest addresses and proxy edges, missing resolver manifests, drifted topic-filtered coverage, observed same-height forks, and replay range-start loading. Pure evaluator regressions cover cursor-ahead/empty-log projection restores, backlog gaps, missing target manifests, and topic drift. The shared drift guard remains covered by both stored-lineage promotion regressions.

Shape

bigname_storage::load_data_completeness performs the reads. The verdict is a pure function over that read plus the runtime watch plan, so it is unit-tested without a database against the shapes that matter: the empty database whose queues are all trivially drained, the crash-looping replay cursor, a lineage gap, a retired target that must not be required to be observed, an empty watch set, a missing replay cursor, trailing observed logs above the canonical head, a latched chain at and short of its backlog target, a pending invalidation, and a dead letter.

Coverage is computed per (chain, address) pair, since load_watched_contracts returns one row per source entry and a target can repeat across source families.

Scope

Database-level only. HTTP route spot-checks and cross-database name-count parity are deliberately deferred and called out as such in the runbook, as is the live tail beyond a latched chain's backlog target. A passing gate is a necessary condition for a cutover, not a sufficient one.

docs/runbooks/data-completeness.md documents each check, how to read each failure, why /v1/status cannot substitute, and why inactive-chain or non-active-profile cursors are advisory.

Tests

CI is green for b8f128f: static analysis, artifacts/smoke, API, core, indexer, worker, conformance, Manager GraphQL compatibility, and the aggregate test gate all passed.

Local verification on the final commit graph passed formatting, workspace compilation, the exact CI clippy command, diff hygiene, and the base-relative Rust file-size gate. Database-backed suites passed 296/296 worker tests, 443/443 storage tests with four test threads, 366/366 runnable indexer tests (three credential/scale tests explicitly ignored), and 53/53 manifest tests. Focused completeness verification passed 64/64 worker tests, 19/19 storage tests, and both stored-lineage topic-drift promotion regressions.

Add `bigname-worker inspect data-completeness`, a read-only command that answers whether a database is complete enough to serve, and exits non-zero with `--fail-on-incomplete` so a promotion or cutover can gate on it. `/v1/status` cannot answer this. It derives `latest_projected_block` from the projection queue and reports the stored canonical checkpoint whenever that queue is empty, so an empty work queue reads as caught up to head. Every cursor in the pipeline measures itself against the previous stage's frontier, so an upstream stall or a watch set that silently excludes targets leaves all of them reporting "done". An empty database and a complete one are indistinguishable to that endpoint. The command checks, in dependency order: the reconciliation frontier is within a tolerance of the stored canonical checkpoint; reconciled lineage is contiguous; every active watched target has at least one non-orphaned code observation; no normalized replay cursor carries a failure reason; each replay cursor has reached its chain's raw-log head; each projection apply cursor has drained; and normalized_events and name_current are non-empty. Watch-set code-observation coverage is the check that carries the gate. The others are relative invariants -- each compares one stage to the stage before it, so all of them stay green while the pipeline faithfully processes an incomplete input. Coverage is the only check that compares what was indexed against what the manifests and active discovery edges declare should be indexed. Against the current Sepolia deployment it reports 352 of 355 active watched targets with no code observation, while every other check passes. Storage exposes `load_data_completeness`; the verdict is a pure function over that read plus the runtime watch plan, unit-tested against the empty-database, crash-looping-cursor, lineage-gap, and unwatched-discovered-target shapes. `docs/runbooks/data-completeness.md` documents each check and how to read a failure. Route-level checks -- HTTP spot-checks and cross-database name-count parity -- are deliberately out of scope here. A passing gate is a necessary condition for a cutover, not a sufficient one.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e864eec1a7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-10T04:18:10Z

+    let lagging_projection_cursors = read
+        .projection_apply_cursors
+        .iter()
+        .filter_map(|cursor| {
+            let max_change_id = cursor.max_change_id?;
+            (cursor.last_change_id < max_change_id).then(|| CursorLag {


Gate on pending projection invalidations

When normalized-event changes have already been scanned, projection_apply_cursors.last_change_id can equal max_change_id while projection_invalidations still contains pending, claimed, retry-delayed, or dead-lettered work; derive_normalized_event_invalidations advances this cursor before the separate apply phase runs. In that state this check reports projection_apply_drained as passing and --fail-on-incomplete can approve a cutover even though projections are still stale, so the gate should also require the projection invalidation/dead-letter queues to be drained.

Useful? React with 👍 / 👎.

Fixed in eca7772. Added two checks, projection_invalidations_drained and projection_no_dead_letters, each requiring the respective table to be empty. A successful apply deletes the projection_invalidations row and an exhausted one moves to projection_invalidation_dead_letters, so a drained derive scan with unapplied or dead-lettered invalidations no longer passes.

chatgpt-codex-connector · 2026-07-10T04:18:10Z

+    let lagging_replay_cursors = read
+        .replay_cursors
+        .iter()
+        .filter(|cursor| cursor.cursor_kind == RAW_FACT_NORMALIZED_EVENTS_CURSOR)
+        .filter_map(|cursor| {


Treat missing replay cursors as incomplete

If a chain has retained raw logs but its raw_fact_normalized_events cursor is missing, this iterator simply produces no lag row, so normalization_caught_up() passes rather than failing the chain. That can happen before catch-up has ever created the cursor or after an incomplete restore, and a database with non-empty projections can then pass the promotion gate despite never normalizing that chain's raw facts; the check should synthesize expected cursor coverage from the raw-log heads or explicitly fail missing cursor rows.

Useful? React with 👍 / 👎.

Fixed in eca7772. The evaluator now requires a raw_fact_normalized_events cursor for every chain that has a canonical raw-log head. A chain with retained canonical raw logs but no cursor fails normalization_caught_up_to_raw_head with a chains_missing_raw_fact_cursor entry instead of passing by absence.

chatgpt-codex-connector · 2026-07-10T04:18:10Z

+        SELECT
+            projection_apply_cursors.cursor_name,
+            projection_apply_cursors.last_change_id,
+            max_change.max_change_id
+        FROM projection_apply_cursors


Fail when the projection apply cursor is absent

Because this query is anchored on projection_apply_cursors, a database with projection_normalized_event_changes rows but no normalized_events_to_projection_invalidations cursor returns an empty cursor list; the evaluator then treats the projection stage as drained. This is the pre-handoff/truncated-cursor shape where continuous apply has not consumed the change log, so the completeness gate should report it as incomplete instead of passing by absence.

Useful? React with 👍 / 👎.

Fixed in eca7772. max(change_id) is now loaded independently of projection_apply_cursors, and the check fails when the change log is non-empty but no apply cursor row exists.

chatgpt-codex-connector · 2026-07-10T04:18:10Z

+        raw_log_head AS (
+            SELECT chain_id, MAX(block_number) AS raw_log_head_block_number
+            FROM raw_logs
+            WHERE canonicality_state <> 'orphaned'::canonicality_state
+            GROUP BY chain_id


Compare replay lag to canonical raw-log head

This head includes any raw log that is not orphaned, including blocks still in observed state, but normalized replay only advances over raw logs whose raw row and lineage are canonical/safe/finalized. During live intake or reconciliation lag, a newly observed block can therefore make normalization_caught_up_to_raw_head fail even though the replay cursor has caught up to every replay-eligible canonical raw fact; use the same canonical filters as the replay selection for this gate.

Useful? React with 👍 / 👎.

Fixed in eca7772. The gate now compares replay progress against a canonical raw-log head computed by joining chain_lineage and filtering both raw_logs.canonicality_state and the lineage state to canonical/safe/finalized, matching the replay selection bounds. The non-orphaned head is retained as a reported-only value.

chatgpt-codex-connector · 2026-07-10T04:18:10Z

+    pub(super) fn watch_set_observed(&self) -> CheckStatus {
+        CheckStatus::from_pass(self.unobserved_targets.is_empty())


Require a non-empty active watch set

When load_watched_contracts returns no active targets, unobserved_targets is also empty, so this check passes vacuously. A restored or candidate database that retained old raw/projection rows but lost or never synced active manifest/watch-plan rows can then pass the cutover gate with no declared watch scope for future indexing, so the coverage check should fail when the active watch set is empty.

Useful? React with 👍 / 👎.

Fixed in eca7772. watch_set_code_observation_coverage now fails when there are zero active watched targets.

chatgpt-codex-connector · 2026-07-10T04:18:10Z

+    let lagging_replay_cursors = read
+        .replay_cursors
+        .iter()
+        .filter(|cursor| cursor.cursor_kind == RAW_FACT_NORMALIZED_EVENTS_CURSOR)
+        .filter_map(|cursor| {


Account for the post-replay backlog cursor

For chains that use closure/dependency replay, the raw_fact_normalized_events cursor intentionally keeps its latched replay target while newer persisted raw logs are normalized by the separate post_replay_live_adapter_backlog cursor. This filter ignores that backlog cursor and compares only the latched raw-fact cursor to the raw-log head, so a healthy database with the backlog fully drained still reports normalization as lagging and blocks the cutover gate.

Useful? React with 👍 / 👎.

Fixed in eca7772. Chains with a post_replay_live_adapter_backlog cursor row are treated as latched: the gate requires both the raw_fact_normalized_events cursor and the backlog cursor to reach their own targets rather than comparing the latched cursor to the raw-log head. The live tail beyond the backlog target has no cursor and is documented as a gate limitation in the runbook.

…tch-set checks Six review findings, four of which let the gate approve an incomplete database and two of which block a healthy one. Projection application was only proven as far as the derive scan. The apply cursor reaching max(change_id) means the scan finished, but the resulting invalidations move through a separate claim/apply queue; a successful apply deletes its projection_invalidations row and an exhausted one moves to projection_invalidation_dead_letters. Add projection_invalidations_drained and projection_no_dead_letters, both requiring an empty table, so a drained scan with unapplied projection writes no longer passes. The projection apply check anchored on projection_apply_cursors with a cross join, so an empty cursor table returned no rows and passed even with a non-empty change log. Load max(change_id) independently and fail when the change log is non-empty but no apply cursor exists. normalization_caught_up_to_raw_head compared replay progress against the non-orphaned raw-log head, which includes observed logs that replay is not eligible to consume; a candidate with trailing observed logs was stuck failing. Compare against the canonical raw-log head instead, mirroring the replay bounds (raw log and its lineage block both canonical/safe/finalized). The non-orphaned head is retained as a reported-only value. The same check iterated existing cursor rows only, so a chain with retained canonical raw logs but no raw_fact_normalized_events cursor passed vacuously. Require the cursor to exist for any chain with a canonical raw-log head. On a chain that ran closure or dependency replay, the raw_fact_normalized_events cursor target is latched permanently below the live head, so comparing it to the head made the gate unable to ever pass there. Detect the latch by a post_replay_live_adapter_backlog cursor row and, for such chains, require the raw-fact and backlog cursors to reach their own targets. The live tail beyond the backlog target has no cursor and is documented as a gate limitation. watch_set_code_observation_coverage passed vacuously at zero active targets, so a manifest-less restore cleared the load-bearing check. Require at least one active watched target. Update the runbook check table, the replay-target and projection sections, the failure guide, and add a caveat for stale cursors from retired deployment profiles.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eca7772bb7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T00:27:27Z

+                .head_lag_blocks
+                .is_some_and(|lag| lag <= self.max_head_lag_blocks)


Reject checkpoints that lag behind retained lineage

When the stored checkpoint is behind the retained canonical lineage (for example canonical_block_number = 100 and lineage_head_block_number = 1000 after a backfill or stale checkpoint writer), head_lag_blocks is negative and this predicate still passes because any negative value is <= max_head_lag_blocks. That lets --fail-on-incomplete approve a database whose serving checkpoint is stale relative to its indexed lineage/projections; require the lag to be non-negative (or otherwise explicitly fail checkpoint-behind-lineage) before treating the reconciliation frontier as caught up.

Useful? React with 👍 / 👎.

Addressed in c2e7a68. The frontier tolerance is now symmetric — |head_lag_blocks| <= max_head_lag_blocks — so the canonical=100 / lineage_head=1000 example (lag −900) fails. A small negative lag within max is tolerated deliberately: reconcile commits canonical lineage and then advances the checkpoint, so a live database is routinely lineage-ahead by a few blocks. A genuinely stale checkpoint writer, being far behind, is beyond tolerance and fails.

chatgpt-codex-connector · 2026-07-13T00:27:27Z

+        WITH known_chains AS (
+            SELECT chain_id FROM chain_checkpoints
+            UNION
+            SELECT DISTINCT chain_id FROM chain_lineage
+        ),


Include manifest chains in frontier checks

This frontier set omits chains that are expected only from active manifests/watch rows, so a restore that retained manifest rows and code observations but lost chain_checkpoints and chain_lineage for that chain produces no ChainCompletenessRow. The frontier, contiguity, and missing-cursor checks then pass vacuously for that expected chain, allowing the promotion gate to approve without any reconciled head metadata; include the active manifest/watch chains here so the null checkpoint/lineage values fail closed.

Useful? React with 👍 / 👎.

Fixed in c2e7a68. The active chain set is now the materialized watch view unioned with active manifest_versions rows directly, and the frontier check synthesizes a failing row (missing_from_storage: true) for any active chain absent from both checkpoints and lineage, so a manifest-declared chain can no longer pass by absence.

chatgpt-codex-connector · 2026-07-13T00:27:27Z

+                MAX(block_number) AS lineage_head_block_number,
+                MIN(block_number) AS lineage_floor_block_number,
+                COUNT(DISTINCT block_number) AS lineage_canonical_block_count


Detect duplicate canonical lineage heights

If reorg repair or a bad restore leaves two non-orphaned hashes at the same block number, COUNT(DISTINCT block_number) still equals head - floor + 1, so reconciliation_lineage_contiguous reports pass even though the canonical/safe/finalized lineage is ambiguous for that height. Because chain_lineage is keyed by (chain_id, block_hash) rather than (chain_id, block_number), the gate should also fail when any canonical block number has more than one retained hash before approving a cutover.

Useful? React with 👍 / 👎.

Fixed in ac75bc3. A per-chain count of block heights carrying more than one non-orphaned canonical/safe/finalized lineage row is loaded, and reconciliation_lineage_contiguous fails when it is non-zero. c2e7a68 adds a DB-backed test seeding two canonical hashes at one height.

chatgpt-codex-connector · 2026-07-13T00:27:27Z

+                MAX(block_number) AS lineage_head_block_number,
+                MIN(block_number) AS lineage_floor_block_number,
+                COUNT(DISTINCT block_number) AS lineage_canonical_block_count


Require lineage to cover declared start blocks

When a candidate has only recent live-tail data for a watched contract whose manifest declares a much earlier start_block, this MIN(block_number) simply becomes the late first retained block and the contiguity check compares only that truncated span through head. With a baseline code observation and some projection content, --fail-on-incomplete can approve a database missing all historical logs and derived state before that floor; the gate should compare the floor or backfill coverage against active watched targets' finite start blocks.

Useful? React with 👍 / 👎.

Fixed in ac75bc3 via reconciliation_history_from_declared_start: each active watched chain's lineage floor must be at or below the earliest finite active_from_block_number across its active targets. c2e7a68 additionally fails a chain whose targets are all open-ended, since no floor can be established.

chatgpt-codex-connector · 2026-07-13T00:27:27Z

+    pub(super) fn content_present(&self) -> CheckStatus {
+        CheckStatus::from_pass(self.normalized_event_count > 0 && self.name_current_count > 0)


Scope projection content check to active datasets

If the database retains normalized_events and name_current rows from another chain/profile/namespace while a newly active watched chain has no projected names, this global count still passes. Since the watch and frontier checks are per active chain, data_complete can approve a cutover where one active manifest's projections are empty as long as any other projection data exists; this should be checked against the active watched dataset rather than repository-wide totals.

Useful? React with 👍 / 👎.

Addressed by the wave-3 rework (ac75bc3): the global content_present count was replaced with active_dataset_non_empty, which derives the expected (chain, namespace) set from active manifest declarations and requires each active pair to have its own non-orphaned normalized_events (and each namespace its own name_current). Repository-wide totals are now report-only, so another chain/profile/namespace's rows can no longer satisfy an empty active dataset. c980675 narrows that expected set further to event-producing manifests. No further change needed here.

…t and history Five review findings from the re-review, three where the gate would approve an incomplete database, one where it would block a healthy one, and one that widens the content check against the declared world. The frontier check accepted a negative head lag. A canonical checkpoint behind the retained lineage head is a stale checkpoint writer or a mixed restore, not a caught-up frontier, so the check now requires the lag to be in `0..=max`. The set of chains came from checkpoints unioned with lineage, so a chain the active watch set declares but that is absent from both tables produced no row and every per-chain check passed by absence. The evaluator now synthesizes a failing frontier row for each active watched chain with no storage row. A duplicate canonical height was invisible: `COUNT(DISTINCT block_number)` cannot see two non-orphaned canonical/safe/finalized rows at one height. A per-chain duplicate-height count is loaded and folded into the contiguity check. A live-tail-only database passed every check, because the contiguity span runs from the retained lineage floor: the truncated span is itself contiguous, cursors catch up over the short raw set, and projections are non-empty. A new `reconciliation_history_from_declared_start` check requires each active watched chain's lineage floor to sit at or below the earliest finite start block its active targets declare; a target with an open-ended start imposes no floor. `projections_non_empty` counted the global tables, so rows from another chain, profile, or namespace satisfied it while a newly active chain had none. It is renamed `active_dataset_non_empty` and scoped: each active watched chain must have `normalized_events` by `chain_id`, and each namespace those chains produce must have `name_current` by namespace, the finest dimension a chain-less name projection has. Update the runbook check table, the new-check and failure sections, and the caveat list.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac75bc3318

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T02:54:17Z

+    let projection_apply_cursor_missing =
+        read.max_projection_change_id.is_some() && read.projection_apply_cursors.is_empty();


Require the apply cursor by name

Fresh evidence after the follow-up: this still treats any row in projection_apply_cursors as proof that continuous apply has a cursor. I checked the projection worker and it reads only normalized_events_to_projection_invalidations (apps/worker/src/projection_apply.rs defines that cursor and derive.rs filters by cursor_name = $1), so a restore that keeps an unrelated/stale cursor row while dropping the real one makes projection_apply_cursor_missing false; if that stale row is at or above max(change_id), --fail-on-incomplete can approve even though the change log has no consumable apply cursor. Key the missing/lag check to the expected cursor name instead of table emptiness.

Useful? React with 👍 / 👎.

Fixed in c980675. projection_apply_cursor_missing and lagging_projection_cursors are now keyed to the normalized_events_to_projection_invalidations cursor name rather than to projection_apply_cursors being non-empty, so a stale or unrelated cursor row can no longer mask the missing real cursor or stand in for it above max(change_id). Covered by stale_apply_cursor_does_not_mask_missing_real_cursor and stale_apply_cursor_behind_is_not_counted_as_lag.

chatgpt-codex-connector · 2026-07-13T02:54:17Z

+        SELECT DISTINCT chain_id, lower(contract_address) AS address
+        FROM raw_code_hashes
+        WHERE canonicality_state <> 'orphaned'::canonicality_state


Match code observations to the active range

For targets whose active_from_block_number is later than zero, collapsing raw_code_hashes to just (chain_id, address) lets a pre-admission observation satisfy coverage. A warmed or restored DB can retain an old code-hash row for the same address but have no baseline observation after the manifest/discovery start for the watched target; watch_set_code_observation_coverage then passes and --fail-on-incomplete can approve a database where the current target was never actually watched. Preserve the observation block number and require it to fall within the watched contract's active range.

Useful? React with 👍 / 👎.

Fixed in c980675. load_observed_code_addresses now carries MAX(block_number) per (chain, lower(address)), and coverage requires an observation at or after the target's active_from_block_number, so a pre-admission observation of a reused address no longer satisfies a target whose declared start is later. Covered by the DB test observed_code_addresses_carry_max_non_orphaned_block and the gate tests observation_below_active_from_does_not_satisfy_coverage / observation_at_active_from_satisfies_coverage.

chatgpt-codex-connector · 2026-07-13T02:54:17Z

+                    let completed = cursor.last_completed_block_number.unwrap_or(-1);
+                    (completed < head).then(|| CursorLag {


Use replay next_block_number for completion

The catch-up and backlog runners consider a replay cursor incomplete while next_block_number <= target_block_number, but this gate only compares last_completed_block_number to the head/target. After a partial restore or cursor inconsistency where last_completed_block_number is at the target but next_block_number still points inside the range, normalization_caught_up_to_raw_head can pass and approve cutover even though the replay worker would continue processing that cursor; include next_block_number in the read and fail unless it has advanced past the applicable target.

Useful? React with 👍 / 👎.

Fixed in c2e7a68. The caught-up determination now reads next_block_number > target_block_number — the same completion authority the catch-up loop uses — instead of the monotonic last_completed_block_number, so a reorg that rewinds next_block_number below the target no longer reads as caught up.

Already handled since c2e7a68. The gate completion helper requires the next_block_number / target_block_number pair to satisfy next > target; last_completed_block_number is reporting detail only (current code). The rewound-cursor regression keeps last_completed at the target while lowering next and verifies the gate fails. No further change was needed in the synthesis merge. CI is green.

…d integrity gaps Thirteen findings from a paired adversarial review of the previous gate commit, plus four design decisions, each implemented at its recommended default. Content and chain authority. The expected content set now comes from active manifest_versions rows, not from observed events, so a chain declared to produce two namespaces fails when one is wholly missing. The active chain set is the watch view unioned with active manifest_versions directly, so a partial restore that lost contract_instance_addresses rows cannot delete a chain from its own expectations. Per-chain gating applies to active chains only; a foreign or retired chain with residual storage rows is reported as an advisory. Content counts exclude orphaned normalized events, and NULL chain_id rows are excluded from the per-chain counts and surfaced as a data-integrity check rather than aborting the read. Replay completion. The caught-up determination reads the cursor's next/target pair, the pipeline's own completion authority, instead of the monotonic last_completed high-water mark. A reorg rewind lowers next below target while last_completed stays high, so a rewound cursor now fails. The head-lag tolerance is symmetric: reconcile commits canonical lineage before advancing the checkpoint, so a small negative lag is a normal live state and only a larger gap in either direction fails. Projection completeness. A new check requires all current projections to have a current_projection_replay_status marker at the newest replay version present, matching the worker's bootstrap handoff; name_current is published first, so requiring only it let a mid-bootstrap candidate pass. The version is read from the data, not hardcoded, so a database built by an older image is judged at its own version. The markers are the authority; per-projection non-empty checks are not added, because a projection can be legitimately empty for a dataset. A separate check verifies the eight deferred normalized_events projection indexes exist, since a fresh replay drops them. History. A chain whose active targets are all open-ended (no finite start block) now fails, since no history floor can be established; finite-start targets are unchanged and open-ended siblings still impose no floor, matching bootstrap's authority. Backfill lifecycle is a per-profile advisory (failed, incomplete, expired-lease counts), not a gate: without coverage-fact reconciliation a failed range cannot be distinguished from one superseded by a retry, and the data-level checks already gate the outcome. Coinbase SQL sample-mode's sparse lineage is documented as a non-promotable state in the runbook rather than special-cased in code. Add DB-backed storage tests for the duplicate-height CTE, the manifest-declared chain missing from checkpoints and lineage, the orphaned and NULL chain_id exclusion, the rewound cursor read, and the deferred indexes. Split the evaluator's report types into a submodule to stay within the file-size gate. Update the runbook check table, the new-check and failure sections, the advisories, and the Coinbase and open-ended caveats.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2e7a682e9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T07:09:19Z

+                chain_id,
+                MAX(block_number) AS lineage_head_block_number,
+                MIN(block_number) AS lineage_floor_block_number,
+                COUNT(DISTINCT block_number) AS lineage_canonical_block_count


Reject disconnected canonical lineage

If a restore or reorg repair leaves exactly one canonical/safe/finalized row at every height but a row's parent_hash does not point to the previous canonical hash, this height-only aggregate still makes reconciliation_lineage_contiguous pass. The storage canonicality model is hash/parent based, so --fail-on-incomplete can approve a chain whose retained canonical branch is internally disconnected even though all downstream raw-log and projection state is anchored to that corrupted lineage.

Useful? React with 👍 / 👎.

Fixed in c980675. load_chain_completeness now counts canonical/safe/finalized rows above the chain's canonical floor whose parent_hash matches no canonical row at the preceding height (disconnected_canonical_parent_count), and reconciliation_lineage_contiguous fails when it is non-zero — so a branch complete by height but broken by hash no longer passes. Covered by the DB test disconnected_canonical_parent_is_counted and the gate test disconnected_canonical_parent_fails_contiguity.

chatgpt-codex-connector · 2026-07-13T07:09:19Z

+        SELECT indexname
+        FROM pg_indexes
+        WHERE schemaname = 'public'
+          AND indexname = ANY($1::TEXT[])


Require deferred indexes to be valid

When a CREATE INDEX CONCURRENTLY restore/rebuild fails after creating the catalog entry, pg_indexes can still list the index even though pg_index.indisvalid is false and PostgreSQL cannot use it. This makes deferred_projection_indexes_present pass and allows a cutover with the replay-critical normalized-event indexes still unusable; join pg_index/pg_class and require the expected indexes to be valid, not just named.

Useful? React with 👍 / 👎.

Fixed in c980675. load_present_deferred_projection_indexes now joins pg_index/pg_class and requires indisvalid and indisready, so an index left invalid by a failed CREATE INDEX CONCURRENTLY (still listed by pg_indexes) is not counted present. Covered by the DB test invalid_deferred_index_is_not_reported_present, which leaves an invalid index behind via a failed concurrent unique build.

chatgpt-codex-connector · 2026-07-13T07:09:19Z

+    let mut active_targets = BTreeMap::<(&str, String), &WatchedContract>::new();
+    for contract in watched_contracts
+        .iter()
+        .filter(|contract| is_active(contract))


Require manifest-declared targets in coverage

Because the expected coverage set is derived only from load_watched_contracts, an active manifest target whose contract_instance_addresses row was lost in a partial restore disappears from active_targets instead of becoming unobserved. If at least one sibling target remains observed, watch_set_code_observation_coverage, history, and content can all pass while a manifest-declared contract was never watched; include the active manifest_contract_instances.declared_address set (or otherwise fail missing address rows) before approving a cutover.

Useful? React with 👍 / 👎.

Fixed in c980675. A new manifest_declared_targets_present check reads active manifest_contract_instances directly and fails on any declared instance with no live contract_instance_addresses row — the case where load_watched_contracts (which reads the address from that row) would otherwise drop the target from the watch view entirely rather than reporting it unobserved. Covered by the DB test manifest_declared_target_without_live_address_is_surfaced and the gate test manifest_declared_target_missing_address_fails.

chatgpt-codex-connector · 2026-07-13T07:09:19Z

+        SELECT DISTINCT chain, namespace
+        FROM manifest_versions
+        WHERE rollout_status = 'active'


Exclude non-event manifests from content expectations

This treats every active manifest row as a (chain, namespace) that must have normalized events, but some active manifests only supply execution/transport metadata rather than event intake; for example the checked-in basenames_execution/basenames_l1_compat Ethereum manifests declare basenames without event sources. A complete mainnet database can therefore fail active_dataset_non_empty on (ethereum-mainnet, basenames) even though those manifests are not expected to produce normalized events, so the expected set should be limited to source families/capabilities that actually write normalized events.

Useful? React with 👍 / 👎.

Fixed in c980675. load_manifest_chain_namespaces now limits the expected content set to active manifests that declare normalized-event outputs (a non-empty abi.events entry with a normalized_events mapping), so execution/transport manifests like basenames_execution and basenames_l1_compat — which declare basenames on ethereum-mainnet without event sources — no longer force active_dataset_non_empty to fail on (ethereum-mainnet, basenames). Covered by the DB test active_event_producing_manifest_chain_namespaces_are_loaded.

chatgpt-codex-connector · 2026-07-13T07:09:19Z

+        SELECT chain_id, namespace, COUNT(*)::BIGINT AS count
+        FROM normalized_events
+        WHERE chain_id IS NOT NULL
+          AND canonicality_state <> 'orphaned'::canonicality_state
+        GROUP BY chain_id, namespace


Scope event counts to active manifest sources

These counts accept any non-orphaned normalized_events row for the same chain and namespace, regardless of whether it was produced by an active manifest/source family. After a manifest version or source-family change, leftover rows from a deprecated manifest can satisfy active_dataset_non_empty while the newly active source has produced zero events, so the promotion gate can approve a database missing the active manifest's derived inputs; join/filter the count against active manifest identities instead of only (chain_id, namespace).

Useful? React with 👍 / 👎.

Real concern, deferring rather than landing a narrower fix that risks a new false-fail. Scoping the content count to active manifest identities via normalized_events.source_manifest_id would drop legitimately-active events that carry a NULL source_manifest_id (repair paths write NULL — normalized_events/upsert/repair.rs), reintroducing exactly the false-fail this PR has been closing. A correct version needs to define what a NULL stamp means for an active event and how a manifest-version bump re-stamps source_manifest_id during re-derivation, so the count tracks the active manifest without failing a complete database mid-lifecycle. Tracking separately.

Fixed in merge commit 09f60c9. load_active_manifest_event_sources derives only active sources that declare normalized output and joins events on the exact source_manifest_id, manifest version, chain, namespace, source family, and declared event kind. Rows from a deprecated or prior manifest identity cannot satisfy the active source; a manifest-version rollout without re-derivation therefore fails intentionally. Covered by active_event_sources_are_counted_by_exact_manifest_identity. CI is green.

…checks Fourth review follow-up, closing six holes surfaced in re-review: - Key projection-apply presence and lag to the normalized_events_to_projection_invalidations cursor name, so a stale or unrelated cursor row left by a restore cannot stand in for the missing real cursor (by masking it or reading as caught up in its place). - Require a code observation within a target's active range (at or after its declared start), so a pre-admission observation of a reused address no longer satisfies watch-set coverage. - Count canonical lineage rows whose parent_hash links to no canonical row at the preceding height, so a branch complete by height but broken by hash fails lineage contiguity. - Require the deferred normalized-event indexes to be valid (indisvalid and indisready), not merely listed by pg_indexes, so an invalid index left by a failed concurrent rebuild is not reported present. - Add manifest_declared_targets_present: an active manifest-declared instance with no live contract_instance_addresses row fails directly instead of vanishing from the watch view when a sibling target remains observed. - Scope the expected content set to active manifests that declare normalized-event outputs, so execution/transport manifests (basenames_execution, basenames_l1_compat) no longer false-fail active_dataset_non_empty on a complete mainnet database. Adds DB-backed storage tests and gate unit tests for each shape and documents the checks in docs/runbooks/data-completeness.md.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c980675bc3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T23:46:22Z

+        SELECT DISTINCT replay_version, projection
+        FROM current_projection_replay_status


Require replay markers to cover the target

This read drops completed_normalized_target_block, so projection_replay_complete only checks that each projection has a marker at the newest version. In a restored/candidate DB where markers were written for an old target but raw_fact_normalized_events has since advanced, and the change log was truncated or never captured those historical rows, --fail-on-incomplete can pass even though the worker would not treat those markers as covering the bootstrap target: both projection_replay_completed and load_current_projection_replay_marker_count require completed_normalized_target_block >= replay_target_block. Include that column and compare it to the normalized replay target before accepting the markers.

Useful? React with 👍 / 👎.

Fixed in merge commit 09f60c9. The storage read now loads completed_normalized_target_block and computes the same target automatic bootstrap requests, max(raw_fact_normalized_events target, chain checkpoint frontier) (loader). The evaluator accepts a marker only when its stored target covers that required target (check). Covered by the DB loader test and projection_replay_markers_below_required_target_fail. CI is green.

chatgpt-codex-connector · 2026-07-13T23:46:22Z

+            SELECT 1 FROM contract_instance_addresses cia
+            WHERE cia.contract_instance_id = mci.contract_instance_id
+              AND cia.deactivated_at IS NULL


Match live address rows to manifest declarations

This existence check accepts any live contract_instance_addresses row for the declared contract_instance_id, even if a partial restore or bad reconciliation left that row on a different chain or address than mv.chain/mci.declared_address. In that case the manifest-declared target is still absent from the actual watch view (which reads cia.address), but manifest_declared_targets_present passes and coverage can be satisfied by observing the wrong address; require the live row to match the manifest chain and declared address before treating the target as present.

Useful? React with 👍 / 👎.

Fixed in merge commit 09f60c9. load_manifest_declared_targets_missing_address now treats a declaration as materialized only when the live row matches the same contract instance, manifest chain, and lowercased declared address. The DB regression test seeds exact, deactivated, wrong-address, and wrong-chain rows and surfaces every case except the exact match (test). CI is green.

chatgpt-codex-connector · 2026-07-13T23:46:22Z

+        match contract.active_from_block_number {
+            Some(start) => {
+                info.finite_min_start = Some(
+                    info.finite_min_start
+                        .map_or(start, |current| current.min(start)),


Do not let address ranges raise manifest starts

Fresh evidence after the declared-start fix: contract.active_from_block_number here is the watch-view effective start; load_watched_contracts computes it as the greater of the manifest start and contract_instance_addresses.active_from_block_number. If a restore keeps the active manifest but recreates the live address row with active_from_block_number = 900 for a manifest start of 100, this loop records 900 as the declared floor, so a lineage floor of 900 passes and --fail-on-incomplete can approve a database missing blocks 100–899. Load the manifest-declared start directly, or fail when the live address row narrows it.

Useful? React with 👍 / 👎.

Fixed in merge commit 09f60c9. Direct manifest targets now load start_block from the active manifest payload independently of contract_instance_addresses (loader). The evaluator unions those direct declarations with watch rows, so the manifest start remains history authority even when a materialized address row has a later effective start. Covered at the storage boundary by a manifest start of 42 plus address start of 900 (test) and in the gate by watched_address_start_does_not_raise_manifest_history_start. CI is green.

Require the projection worker's named cursor and code observations within active target ranges. Keep manifest declarations and event-producing source identities authoritative across partial restores. Reject disconnected canonical lineage and invalid deferred indexes before promotion.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09f60c9ffd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-14T02:39:57Z

+    let projection_replay_version = read
+        .projection_replay_markers
+        .iter()
+        .map(|marker| marker.replay_version)
+        .max();


Require current projection replay version

When a candidate DB has complete markers from an older replay version, this takes that older max() version as authoritative and projection_replay_complete can pass. I checked the worker handoff path in apps/worker/src/automatic_projection_replay.rs: load_current_projection_replay_marker_count filters on replay::CURRENT_PROJECTION_REPLAY_VERSION, so the current worker would still run bootstrap replay for exactly that DB; --fail-on-incomplete can therefore approve a cutover before projections have been rebuilt for the code that will serve it. Compare against the current replay version rather than the newest version present.

Useful? React with 👍 / 👎.

Fixed in d08b166. The evaluator now filters completion markers on CURRENT_PROJECTION_REPLAY_VERSION; the newest stored version remains diagnostic output only (check). Complete markers from the prior replay version now fail in complete_markers_at_older_version_fail. CI is green.

chatgpt-codex-connector · 2026-07-14T02:39:57Z

+        SELECT DISTINCT
+            manifest.chain,
+            manifest.source_family,
+            lower(declaration.declared_address) AS address,
+            manifest_range.start_block AS active_from_block_number


Include proxy implementations in declared targets

For manifests that declare a proxy implementation, this declared-target read only emits the proxy declared_address, and the missing-address read below repeats the same shape. The manifest sync code seeds a separate implementation_contract_instance_id/declared_implementation_address and the source graph turns it into a watched proxy-implementation edge; if a partial restore drops that implementation's live address row, load_watched_contracts loses the edge and the direct manifest union never adds it, so the coverage and presence checks can pass while a manifest-declared implementation target was never observed. Include implementation address rows in these declared-target/missing-address reads.

Useful? React with 👍 / 👎.

Fixed in d08b166. Active manifest payload targets now include each non-null proxy implementation with the declaration start block (target expansion), and the named presence check resolves that target through implementation_contract_instance_id plus its matching live address (materialization check). manifest_proxy_implementation_is_a_declared_target covers the missing and restored implementation-address shapes. CI is green.

chatgpt-codex-connector · 2026-07-14T02:39:57Z

+        FROM manifest_versions manifest
+        JOIN manifest_contract_instances declaration
+          ON declaration.manifest_id = manifest.manifest_id


Fail when manifest contract rows are missing

This makes the fallback authority depend on manifest_contract_instances, the same materialized table that feeds load_watched_contracts. If a restore keeps an active manifest_versions.manifest_payload but loses one of these materialized declaration rows, the target disappears from both the watch view and this direct-manifest union, so coverage, declared-target presence, and history can all shrink instead of reporting the missing manifest contract. Parse the active manifest payload or add a check that every payload root/contract has a matching manifest_contract_instances row before approving cutover.

Useful? React with 👍 / 👎.

Fixed in d08b166. Direct targets are now parsed from active manifest_versions.manifest_payload roots and contracts rather than derived from manifest_contract_instances (payload authority, target read). The separate presence query fails when the matching materialized declaration is absent. manifest_payload_target_survives_missing_contract_instance_row verifies the target remains in the coverage/history universe and is reported as a materialization gap. CI is green.

chatgpt-codex-connector · 2026-07-14T02:39:57Z

+                COALESCE(canonical_block_number, -1),
+                COALESCE(safe_block_number, -1),
+                COALESCE(finalized_block_number, -1)
+            )), -1) AS target_block
+            FROM chain_checkpoints


Scope replay target to active chains

When the database still has chain_checkpoints for a retired or foreign chain with a higher head than the active serving set, this unscoped MAX raises projection_replay_required_target_block; the evaluator then treats otherwise complete projection markers as below target and --fail-on-incomplete blocks. This conflicts with the surrounding evaluator logic that reports foreign chains as an advisory rather than a gate, so the replay target should ignore inactive/foreign checkpoint rows or compute it from the active chain set.

Useful? React with 👍 / 👎.

Verified against the writer; target scoping was intentionally not changed. Automatic replay takes the unscoped maximum raw-fact cursor target (writer cursor query) and the unscoped maximum persisted chain checkpoint (writer checkpoint query), then uses the greater value as the global projection marker target (handoff). Scoping only the inspector could approve a marker below the target the writer requests. d08b166 documents that distinction and adds a retired-chain checkpoint regression (runbook, test). CI is green.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d08b16694c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-14T06:47:35Z

+    for target in &read.manifest_declared_targets {
+        active_target_entries.insert((
+            target.chain.clone(),
+            target.address.to_ascii_lowercase(),
+            target.source_family.clone(),
+            target.active_from_block_number,
+        ));


Preserve discovered targets when address rows vanish

If a partial restore keeps an active discovery_edges row but loses or deactivates the target contract_instance_addresses row, load_watched_contracts drops that edge because it joins only live address rows, and this fallback only re-adds direct manifest payload targets. With any sibling target still observed, the coverage/history checks can pass while a discovered resolver or subregistry is absent from the watch set, so the gate should also load active discovery-edge endpoints or fail missing address rows for them.

Useful? React with 👍 / 👎.

Fixed in 547dc2c. The completeness read now loads active discovery-edge endpoints independently of the watched-contract address join, and the new discovery_targets_present check fails when an endpoint lacks a live address on the edge chain (query, DB regression). CI is green.

chatgpt-codex-connector · 2026-07-14T06:47:35Z

+                && match read.projection_replay_required_target_block {
+                    Some(required) => marker
+                        .completed_normalized_target_block
+                        .is_some_and(|completed| completed >= required),


Account for apply-drained blocks after replay

On a live database after bootstrap, current_projection_replay_status records the replay target that was rebuilt, while later normalized changes are kept current by the projection apply cursor rather than by advancing these markers; the automatic worker also skips bootstrap once the apply cursor exists and current-version markers exist. When the normalized replay/checkpoint target advances but projection_apply_drained has consumed the subsequent change log, this target comparison still treats every marker as missing and blocks --fail-on-incomplete for an otherwise fully applied serving database.

Useful? React with 👍 / 👎.

Fixed in 547dc2c. Replay-marker target coverage is now required only before the durable normalized_events_to_projection_invalidations cursor exists; after handoff, current-version marker presence plus the separately drained apply/change/invalidation state is authoritative (evaluation, regression). CI is green.

chatgpt-codex-connector · 2026-07-14T06:47:35Z

+            JOIN contract_instance_addresses address
+              ON address.contract_instance_id = CASE target.target_kind
+                  WHEN 'declaration' THEN declaration.contract_instance_id
+                  ELSE declaration.implementation_contract_instance_id
+              END


Reject missing proxy implementation edges

When a partial restore keeps the proxy implementation's manifest_contract_instances/contract_instance_addresses rows but drops the managed discovery_edges proxy-implementation edge, this presence check still passes because it only verifies the live address row. The runtime watch view loads proxy implementations through discovery_edges, so coverage can be satisfied by the direct-manifest fallback while the actual watch plan no longer contains the implementation target; require the managed edge as well before approving cutover.

Useful? React with 👍 / 👎.

Fixed in 547dc2c. manifest_declared_targets_present now requires the exact active managed proxy-to-implementation edge, including source manifest, discovery source, admission, and endpoint identities (read, DB regression). CI is green.

chatgpt-codex-connector · 2026-07-14T06:47:35Z

+        .iter()
+        .filter(|chain| active_chains.contains(&chain.chain_id))
+        .filter(|chain| chain.canonical_raw_log_head_block_number.is_some())
+        .filter(|chain| !chains_with_raw_fact_cursor.contains(chain.chain_id.as_str()))


Fail when retained raw logs disappear

If a restore keeps normalized_events, projections, lineage, and code observations but loses the underlying raw_logs, the canonical raw-log head is NULL; this filter then skips both the missing-cursor check and the lag comparison, so a database whose raw facts can no longer be replayed or reorg-repaired can still pass as long as derived rows remain non-empty. For active event-producing sources with retained normalized events, the gate should require corresponding retained canonical raw-log coverage instead of treating an absent raw-log head as complete.

Useful? React with 👍 / 👎.

Fixed in 547dc2c. The new active_raw_facts_retained check requires each matching raw-log-backed active normalized event to retain its exact canonical/safe/finalized raw log; persisted synthetic block-boundary events are excluded from this log-specific predicate (read, DB regression). CI is green.

Correction to the earlier reply: the raw-log-specific predicate introduced in 547dc2c was superseded by 8313d65 after rechecking the documented minimal-retention contract. raw_logs are compactable staging and therefore are not a promotion requirement. The current invariant is exact serving-event lineage, including synthetic boundaries (query, retention regression). CI is green on 92087f7.

chatgpt-codex-connector · 2026-07-14T06:47:35Z

+    let failed_replay_cursors = read
+        .replay_cursors
+        .iter()
+        .filter(|cursor| cursor.last_failure_reason.is_some())
+        .map(cursor_label)


Ignore failed replay cursors on inactive chains

When a database retains a failed normalized_replay_cursors row for a retired or foreign chain, this unscoped collection still makes normalization_no_failure fail even though the frontier and lag checks deliberately scope per-chain gating to active_chains and report foreign chains as advisory. Filter failures to active chains as well, otherwise --fail-on-incomplete can block a healthy active serving set because of stale replay state outside the manifest/watch scope.

Useful? React with 👍 / 👎.

Fixed in 547dc2c. Failure, lag, latch, and raw-fact cursor presence now gate only cursors for the active chain set and inferred active deployment profile; other cursor rows remain visible in advisories.ignored_replay_cursors (evaluation, regression). CI is green.

chatgpt-codex-connector · 2026-07-14T06:47:35Z

+    let chains_with_raw_fact_cursor = read
+        .replay_cursors
+        .iter()
+        .filter(|cursor| cursor.cursor_kind == RAW_FACT_NORMALIZED_EVENTS_CURSOR)
+        .map(|cursor| cursor.chain_id.as_str())
+        .collect::<BTreeSet<_>>();


Key replay cursor coverage by deployment profile

Because this collapses raw-fact cursor presence to just chain_id, any cursor row for the same chain masks the absence of the raw_fact_normalized_events cursor for another deployment profile, even though the replay cursor key includes deployment_profile. In a cutover database that keeps an old profile's caught-up cursor while dropping the profile that will actually serve, chains_missing_raw_fact_cursor stays empty and the lag check can pass despite the live profile never normalizing that chain's retained raw logs; key the expected cursor coverage by deployment profile or make the inspected profile explicit.

Useful? React with 👍 / 👎.

Fixed in 547dc2c. The active profile is inferred from the active manifest corpus with the same mainnet / sepolia-dev classification used by replay admission, and cursor failure, lag, latch, and presence checks now require both that profile and an active chain (profile read, evaluation, regressions). CI is green.

Correction to the earlier reply: sepolia-dev was not the writer's cursor profile for manifests/sepolia. 8313d65 replaces that inference with shared authority that returns sepolia for the active Sepolia corpus (profile loader); replay admission and inspection both consume it (writer admission). The sepolia replay integration passes against the checked-in epoch shape (test). CI is green on 92087f7.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 547dc2cfc2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-14T10:18:38Z

+                AND declaration.declaration_kind = target.declaration_kind
+                AND declaration.declaration_name = target.declaration_name
+                AND lower(declaration.declared_address) = target.declared_address
+                AND address.deactivated_at IS NULL


Require manifest address rows to remain open

When an active manifest target's contract_instance_addresses row has active_to_block_number set but deactivated_at is still NULL, this existence check treats the materialized address as present. The evaluator only treats watched contracts as active when active_to_block_number is None, then re-adds the direct manifest target and can satisfy coverage from an old code observation, so --fail-on-incomplete can pass even though the runtime watch view no longer watches that manifest-declared target.

Useful? React with 👍 / 👎.

Fixed in 8313d65. Confirmed: an active manifest declaration now requires a matching address row with both deactivated_at IS NULL and active_to_block_number IS NULL (query). The DB regression verifies that a bounded row is reported missing while the exact open row passes (test). CI is green on 92087f7.

chatgpt-codex-connector · 2026-07-14T10:18:38Z

+                AND edge.discovery_source = 'manifest_declared_proxy'
+                AND edge.source_manifest_id = target.manifest_id
+                AND edge.admission = 'manifest_declared'
+                AND edge.deactivated_at IS NULL


Require proxy implementation edges to remain open

When a manifest-declared proxy implementation has a matching discovery_edges row with a finite active_to_block_number but NULL deactivated_at, this check accepts the edge as present. That closed edge makes load_watched_contracts return the implementation with a finite effective active_to_block_number, which the completeness evaluator excludes from the active watch view; a historical code observation can then let coverage pass while future implementation logs are not actually watched.

Useful? React with 👍 / 👎.

Fixed in 8313d65. Confirmed: the exact managed proxy_implementation edge must now have both deactivated_at IS NULL and active_to_block_number IS NULL (query). The proxy regression closes the edge at block 99 and verifies the implementation is reported missing from current authority (test). CI is green on 92087f7.

chatgpt-codex-connector · 2026-07-14T10:18:38Z

+    if rows.iter().all(|(chain, deployment_epoch)| {
+        chain.ends_with("-sepolia") && deployment_epoch.ends_with("_sepolia_dev")
+    }) {
+        return Ok(Some("sepolia-dev".to_owned()));


Match the Sepolia replay cursor profile

For the documented Sepolia runtime rooted at manifests/sepolia, the automatic normalized-replay writer uses deployment_profile_from_manifest_root, which derives the cursor profile as sepolia, but this inspector infers sepolia-dev from the active manifest rows. That makes cursor_is_active ignore the writer's healthy sepolia/.../raw_fact_normalized_events cursors and --fail-on-incomplete blocks a complete Sepolia cutover; use the same profile authority as the writer or accept the writer's stored profile for this corpus.

Useful? React with 👍 / 👎.

Fixed in 8313d65. Confirmed: the checked-in Sepolia corpus maps to the writer cursor profile sepolia, not sepolia-dev. Replay admission and completeness inspection now share load_active_manifest_deployment_profile, which returns sepolia for the active Sepolia epochs (authority, writer admission). The indexer integration exercises a sepolia replay request against that corpus (test). CI is green on 92087f7.

chatgpt-codex-connector · 2026-07-14T10:18:38Z

+         AND raw.canonicality_state IN (
+             'canonical'::canonicality_state,
+             'safe'::canonicality_state,
+             'finalized'::canonicality_state
+         )


Join retained raw logs through canonical lineage

If a restore keeps a canonical raw_logs row but loses or orphans the matching chain_lineage row, this join still treats the raw fact as retained. The replay head calculation in this same reader requires both the raw log and lineage row to be canonical/safe/finalized, so active_raw_facts_retained can pass even though normalized replay and reorg repair no longer have a canonical block anchor for that serving event.

Useful? React with 👍 / 👎.

Addressed in 8313d65, with the invariant narrowed after checking retention authority. The broken-anchor finding is real, but retained raw_logs cannot be mandatory because minimal retention may compact that staging family. The gate now requires every matching serving event—including synthetic boundaries—to resolve by exact chain/hash/number to retained canonical/safe/finalized lineage (query). The DB regression covers missing, orphaned, and restored exact lineage without requiring compactable raw logs (test). CI is green on 92087f7.

chatgpt-codex-connector · 2026-07-14T10:18:38Z

+        WHERE source_manifest.rollout_status = 'active'
+          AND edge.deactivated_at IS NULL
+          AND edge.edge_kind <> 'migration'


Reject closed discovery edges in presence checks

Filtering discovery edges only by deactivated_at IS NULL lets an edge with a finite active_to_block_number remain authoritative for this check while the evaluator drops the corresponding watched contract because its effective active_to_block_number is non-null. With a live address row and any sibling observed target, discovery_targets_present and coverage can pass even though the discovered resolver/subregistry is no longer in the runtime watch plan for future indexing.

Useful? React with 👍 / 👎.

Addressed in 8313d65. The current-authority mismatch was confirmed, but bounded discovery edges are legitimate retained history rather than corrupt rows. discovery_targets_present now considers only open non-migration edges and requires their target address to remain open; bounded edges are excluded from the current requirement (query). The DB regression covers both an open edge with a closed address (failure) and a bounded historical edge (excluded) (test). CI is green on 92087f7.

chatgpt-codex-connector · 2026-07-14T10:18:38Z

+            lower(contract_address) AS address,
+            MAX(block_number) AS max_observed_block_number
+        FROM raw_code_hashes
+        WHERE canonicality_state <> 'orphaned'::canonicality_state


Anchor code observations to retained lineage

When a partial restore keeps a non-orphaned raw_code_hashes row but loses or orphans the row's block in chain_lineage, this read still counts it as covering the watched target. Because watch_set_code_observation_coverage is the load-bearing check, a target can pass coverage from a code observation that no retained non-orphaned block anchor can prove or reorg-repair; join the observation to retained lineage before treating it as observed.

Useful? React with 👍 / 👎.

Fixed in 8313d65. Confirmed: code observations now join chain_lineage on exact chain, block hash, and block number, and both rows must remain non-orphaned before the observation can cover a target (query). The DB regression shows that orphaned, missing-anchor, and wrong-anchor observations do not advance the loaded coverage height (test). CI is green on 92087f7.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2dfd68a05a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-14T11:26:56Z

+         AND event.namespace = source.namespace
+         AND event.source_family = source.source_family
+         AND event.event_kind = ANY(source.normalized_event_kinds)
+         AND event.canonicality_state <> 'orphaned'::canonicality_state


Require serving-canonical events for active content

When a matching active normalized event is still observed while its lineage row is canonical/safe/finalized (for example after a restore that lost the event canonicality promotion), this join counts it as present and the lineage check passes. The projection/apply readers only consume canonical/safe/finalized normalized_events, so --fail-on-incomplete can approve a source that has no serving projection input; use the same canonical-state filter here or report observed rows as incomplete.

Useful? React with 👍 / 👎.

Fixed in 92087f7. Confirmed: the active-source join now filters the normalized event itself to canonical/safe/finalized before counting it, in addition to requiring the exact serving-canonical lineage anchor (query). The DB regression seeds an observed active event on canonical lineage and verifies its active-source count remains zero (test). CI is green.

chatgpt-codex-connector · 2026-07-14T11:26:56Z

+          AND NOT EXISTS (
+              SELECT 1
+              FROM contract_instance_addresses address
+              WHERE address.contract_instance_id = edge.to_contract_instance_id
+                AND address.chain_id = edge.chain_id
+                AND address.deactivated_at IS NULL
+                AND address.active_to_block_number IS NULL
+          )


Preserve discovery-edge starts in coverage

If an open discovery edge starts earlier than its live address row (for example a partial restore recreates the address row at block 900 for an edge discovered at block 100), this existence check passes and the evaluator only sees load_watched_contracts' later effective start. Coverage and history can then pass from block 900 while logs from 100–899 for that discovered target were never watched; fail when the address row narrows an open edge or carry the edge start into the expected target set.

Useful? React with 👍 / 👎.

Fixed in 92087f7. Confirmed: an open discovery target is present only when its address is open and its start is absent or no later than the edge start. A finite address start cannot satisfy an unknown edge start, so unknown history fails closed (query). The DB regression covers edge 100/address 900 (failure), address 50 (pass), unknown edge/finite address (failure), and both starts unknown (pass) (test). CI is green.

chatgpt-codex-connector · 2026-07-14T11:26:56Z

+        )
+        SELECT
+            known_chains.chain_id,
+            chain_checkpoints.canonical_block_number,


Validate checkpoint hashes against lineage

When a restore leaves canonical_block_number at the lineage head but canonical_block_hash points to a missing, orphaned, or different block at that height, this read still reports zero head lag because it ignores the hash. The checkpoint writer uses that stored hash as the branch anchor for later promotion/reorg work, so --fail-on-incomplete can approve a database whose next checkpoint update is anchored to a broken branch; join the checkpoint hash to retained canonical lineage or expose the mismatch as a failed frontier.

Useful? React with 👍 / 👎.

Fixed in 92087f7. Confirmed: the completeness read now resolves the checkpoint's exact (chain_id, canonical_block_hash, canonical_block_number) against canonical/safe/finalized lineage and exposes that branch-anchor result (query). The frontier requires both that match and the symmetric lag bound (evaluation). DB and pure regressions cover a wrong same-height hash, the exact canonical hash, an orphaned exact hash, and zero numeric lag with a mismatched hash (storage test, worker test). CI is green.

chatgpt-codex-connector · 2026-07-14T11:26:56Z

+        active_namespaces_without_names,
+        normalized_events_null_chain_id_count: read.normalized_events_null_chain_id_count,
+        missing_deferred_projection_indexes,
+        backfill_advisory: read.backfill_lifecycle.clone(),


Gate on durable backfill coverage facts

With a candidate restored from lineage/raw/derived tables but missing backfill_coverage_facts, this remains advisory-only, so --fail-on-incomplete can pass even though stored-lineage promotion later rejects the same chain because watched log-producing intervals cannot be proven fetched (stored_path_has_required_raw_fact_coverage calls find_uncovered_watched_tuples). The surrounding checks prove a lineage span and some content, not that every active watched tuple over that span had raw-log fetch coverage; add a coverage-fact reconciliation check for active intervals or fail when required facts are absent.

Useful? React with 👍 / 👎.

Fixed in 92087f7 with the scope matched to stored-lineage writer authority. New named check stored_lineage_backfill_coverage reconciles retained intervals identified by persisted bounded-backfill jobs—including incomplete/failed crash residue—using the same find_uncovered_watched_tuples anti-join and shared 131072-block chunk size as promotion. Facts from a later successful retry satisfy the interval; ordinary provider-fetched live lineage with no persisted bounded-backfill job remains exempt (check, all-job read). The DB regression verifies missing facts fail even after the checkpoint consumes the range and exact retry facts clear the gap; the pure evaluator regression proves a gap fails data_complete (DB test, evaluator test). CI is green.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92087f7dc8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+                AND address.deactivated_at IS NULL
+                AND address.active_to_block_number IS NULL
+                AND address.chain_id = target.chain
+                AND lower(address.address) = target.address


Reject address rows that narrow manifest starts

Fresh evidence after the earlier declared-start fix: this presence check still accepts an open address row whose active_from_block_number is later than the manifest payload's finite target.start_block. In that restore shape, load_watched_contracts and the backfill coverage anti-join use GREATEST(manifest_start, address_start), so a row recreated at block 900 for a manifest start of 100 can pass manifest presence, history, and code-observation coverage while the actual watch/coverage interval starts at 900 and never proves logs from 100–899. Require the live address row not to narrow the payload start, or report the target missing.

Useful? React with 👍 / 👎.

Fixed in b8f128f. A live address now satisfies manifest presence only when its start is absent or no later than the payload target start; SQL NULL semantics also reject a finite materialization start for an unknown payload start (query). DB regressions cover both finite 100/900 and unknown/900 narrowing, plus the range-preserving repairs (tests, matrix). CI is green.

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+                AND edge.deactivated_at IS NULL
+                AND edge.active_to_block_number IS NULL


Reject proxy edges that narrow implementation starts

Fresh evidence after the proxy-edge openness fix: the managed proxy_implementation edge is accepted when it is open, but its active_from_block_number can still be later than the manifest implementation's finite target.start_block. Because implementations enter the runtime watch view through this discovery edge, a restored edge starting at 900 for a payload start of 100 makes the watch/backfill interval begin at 900 even though the direct manifest target keeps history/code checks green for 100, so logs from 100–899 can be skipped without failing the gate.

Useful? React with 👍 / 👎.

Fixed in b8f128f. The exact managed proxy edge now must be open and start no later than the manifest implementation target, with an open-ended edge required when the payload start is unknown (query). The proxy DB regression changes the edge from start 17 to 99, verifies the gap is reported, then repairs it to 10 before retaining the existing bounded-edge check (test). CI is green.

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+            let max_change_id = read.max_projection_change_id?;
+            (cursor.last_change_id < max_change_id).then(|| CursorLag {


Reject apply cursors beyond the change log

When a partial restore keeps projection_apply_cursors.last_change_id from the source DB but truncates projection_normalized_event_changes below that value (or removes it entirely while the cursor is nonzero), this comparison treats the cursor as drained because it only fails on last_change_id < max_change_id. The projection apply worker scans WHERE change_id > last_change_id, so it would skip all restored change rows at or below the stale cursor and never enqueue the missing invalidations; fail when the cursor is ahead of the retained change-log high watermark instead of approving the cutover.

Useful? React with 👍 / 👎.

Fixed in b8f128f. The named apply cursor is now compared in both directions against the retained high-water mark, treating an empty log as zero; a cursor above either a lower maximum or an empty log fails projection_apply_drained (evaluation, regressions). This matches the derive writer, which scans only change_id > last_change_id (writer). CI is green.

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+                  'ens_v2_registry_l1',
+                  'basenames_base_registry'
+              )
+              OR target_manifest.manifest_id IS NOT NULL


Fail when resolver target manifests vanish

For an open resolver discovery edge from an active registry manifest, losing the active resolver target manifest_versions row makes this predicate drop the edge from discovery_targets_present instead of reporting the missing manifest authority. The runtime watch view uses the same resolver-target-manifest condition, so a partial restore can keep the discovery edge and address/code observation but remove the resolver manifest, shrinking coverage and content expectations while future resolver logs are no longer watched; surface that missing target manifest as an incomplete discovery edge.

Useful? React with 👍 / 👎.

Fixed in b8f128f. A separate read now retains open resolver edges from active registry manifests as authority and reports a missing active resolver target manifest using the same namespace, chain, deployment epoch, and registry-to-resolver family mapping as the runtime watch view (query). The result is part of discovery_targets_present; the DB regression verifies the missing manifest fails while adding the matching active target manifest clears it (test). CI is green.

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+                let uncovered = find_uncovered_watched_tuples(
+                    pool,
+                    &chain.chain_id,
+                    chunk_from,
+                    chunk_through,


Recheck topic drift before trusting coverage facts

For retained bounded-backfill spans, this only anti-joins current watched tuples against existing backfill_coverage_facts, but stored-lineage promotion first verifies that the active manifest ABI topic sets have not drifted before it trusts those facts. If a manifest adds a topic after a family-scoped fact was written, the old fact can still contain the interval and this gate passes, while the promotion writer would refuse because logs for the new topic were never proven fetched; run the same topic-drift guard before accepting the coverage rows.

Useful? React with 👍 / 👎.

Fixed in b8f128f. The exact topic-set drift guard was moved into shared storage authority (guard); stored-lineage promotion still calls it (writer), and completeness now calls the same function before any retained bounded-backfill facts are trusted (inspector). A DB regression changes a completed topic-filtered job to a stale persisted set and confirms facts no longer pass; both writer topic-drift regressions also remain green. CI is green.

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+                SELECT chain_id, block_number
+                FROM canonical_lineage
+                GROUP BY chain_id, block_number
+                HAVING COUNT(*) > 1


Reject non-orphaned same-height forks

This duplicate-height check only groups canonical/safe/finalized rows, so a retained losing branch left in observed state at the same height is invisible as long as exactly one row is canonical. The stored-lineage promotion path rejects any non-orphaned same-height fork before accepting a path, so --fail-on-incomplete can approve a database whose branch repair has not reached the same authority the writer requires; count non-orphaned alternate hashes at retained canonical heights or expose them as a failed frontier.

Useful? React with 👍 / 👎.

Fixed in b8f128f. The duplicate-height CTE now anchors each retained canonical/safe/finalized height and joins every same-height sibling whose state is not orphaned, failing when more than one distinct hash remains (query). The DB regression uses one canonical row plus one observed sibling and verifies the height is counted (test). This matches the promotion refusal for non-orphaned same-height forks. CI is green.

chatgpt-codex-connector · 2026-07-14T12:24:09Z

+                let key = (cursor.deployment_profile.as_str(), cursor.chain_id.as_str());
+                if latched_keys.contains(&key) {
+                    replay_complete_lag(cursor)


Verify backlog cursor starts at replay target

For a latched chain, merely seeing a post_replay_live_adapter_backlog row makes the raw-fact cursor complete against its own old target instead of the raw-log head. If a restore leaves a completed backlog cursor whose range_start_block_number is later than raw_fact_normalized_events.target_block_number + 1, the writer's backlog scan also starts late and blocks in that gap are never normalized, yet this gate passes because it does not read or compare the backlog start; load the start and fail when it leaves a gap after the replay target.

Useful? React with 👍 / 👎.

Fixed in b8f128f. Completeness now loads range_start_block_number for replay cursors (read) and requires a post-replay backlog start at or before the raw-fact target plus one; a later start or missing/overflowed raw target fails with a :range_start lag (evaluation). This mirrors the writer seed at replay_target + 1 (writer); the evaluator regression covers a completed cursor starting at 1500 after target 1000 (test). CI is green.

Anchor cutover expectations to the selected manifest corpus and every current projection family, with retention-aware raw-log verification. Share replay latch and lineage gap policy with writers so healthy idle, quiet closure-replay, pre-activation, and mid-reconcile states are evaluated against their actual completion authority.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe8bad3fee

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-15T01:14:06Z

+            CROSS JOIN LATERAL jsonb_array_elements(
+                COALESCE(manifest.manifest_payload #> '{abi,events}', '[]'::JSONB)
+            ) abi_event
+            CROSS JOIN LATERAL jsonb_array_elements_text(
+                COALESCE(abi_event -> 'normalized_events', '[]'::JSONB)
+            ) normalized_kind(event_kind)


Include adapter-declared reverse events in content checks

This builds the active event-source set only from manifest ABI normalized_events, but the active mainnet ens_v1_reverse_l1 manifest has no abi.events while the ens_v1_reverse_claim adapter emits ReverseChanged. In a mainnet cutover where reverse normalized events or primary_names_current are missing but other ENS projections are non-empty, this source is absent from active_manifest_event_sources, so the active-content, lineage, raw-log, and projection-content checks can all pass without proving the declared reverse-claim intake was projected.

Useful? React with 👍 / 👎.

Fixed in 99d9607. Adapters now declare their emitted normalized event kinds alongside the adapter (ens_v1_reverse_claim and the block-derived builders), and the active event-source universe unions those declarations with the manifest abi events, so the mainnet reverse family carries content, lineage-anchor, and primary_names_current expectations. Covered by checked_in_mainnet_reverse_adapter_requires_primary_name_content and adapter_declared_kinds_extend_active_event_source_universe.

chatgpt-codex-connector · 2026-07-15T01:14:06Z

+        (cursor.last_change_id > retained_change_high_watermark)
+            .then_some(cursor.last_change_id - retained_change_high_watermark)
+    });
+    let projection_replay_target_coverage_required = expected_projection_cursor.is_none();


Require replay coverage when the change log is not authoritative

Fresh evidence after the ahead-of-log fix: any retained apply cursor disables replay-marker target coverage, even a zero cursor paired with an empty or truncated projection_normalized_event_changes table. In a partial restore that keeps current-version replay markers and non-empty stale projections but loses post-bootstrap change rows, projection_apply_drained treats the empty log as drained and this line lets old replay markers pass below the current normalized target, so --fail-on-incomplete can approve projections that the worker will not rebuild or invalidate.

Useful? React with 👍 / 👎.

Fixed in 99d9607. The marker-coverage waiver now requires the apply cursor to be corroborated by a non-empty projection change log; applied change rows are retained outside the rederive path, so a cursor over an empty or truncated log is treated as a restore artifact and no longer waives target coverage. Covered by empty_projection_change_log_does_not_waive_marker_target_coverage.

chatgpt-codex-connector · 2026-07-15T01:14:06Z

+        "permissions_current" | "record_inventory_current" => {
+            let count = load_global_count(pool, projection).await?;
+            (
+                "global",
+                (count > 0)
+                    .then_some(ProjectionScopeCount {
+                        scope: "global".to_owned(),
+                        count,
+                    })
+                    .into_iter()
+                    .collect(),
+            )


Scope resource projections to active chains

When active sources declare permission or record-inventory events for one chain but permissions_current or record_inventory_current only retain rows for another chain, this global count still marks the expected scope present. Both projections are rebuilt per resource_id, with target resources resolved through resources.chain_id, so a partial restore can pass current_projection_content_present while the active chain's resource projections are empty; group these tables through their resources instead of accepting any row globally.

Useful? React with 👍 / 👎.

Fixed in 99d9607. permissions_current and record_inventory_current are now counted through their resources' chains, and each active chain declaring the feeding kinds carries its own expectation, so rows retained from another chain no longer satisfy an active chain. Covered by resource_projections_require_rows_on_each_expected_chain.

chatgpt-codex-connector · 2026-07-15T01:14:06Z

+            RAW_FACT_NORMALIZED_EVENTS_CURSOR => {
+                if closure_replay_chains.contains(cursor.chain_id.as_str()) {
+                    replay_complete_lag(cursor)
+                } else {


Require a backlog cursor for latched tail logs

For closure/dependency replay chains this accepts a completed raw_fact_normalized_events cursor without checking whether canonical raw logs exist above its latched target. The writer creates a post_replay_live_adapter_backlog cursor whenever load_backlog_target_block finds retained canonical logs at target + 1 or later, so a restore that loses that backlog cursor can pass here while those tail blocks were never swept into normalized events; require the backlog cursor when the canonical raw-log head is above the raw-fact target.

Useful? React with 👍 / 👎.

Fixed in 99d9607. A quiet latched chain is accepted only when the canonical raw-log head is at or below the latched raw-fact target; canonical logs above the target require the backlog cursor and its own completion again, mirroring load_backlog_target_block's creation condition. Covered by latched_chain_with_tail_logs_and_no_backlog_cursor_fails and quiet_latched_chain_at_raw_log_head_needs_no_backlog_cursor.

Count projection content through serving filters while retaining raw diagnostics, and scope resource-backed projections by their resource chains. Require tail-log-backed replay backlogs and corroborated projection change-log handoff, then include adapter-owned event kinds in active-source completeness expectations. Document the operational authority rules and cover the restore, foreign-residue, orphan-anchor, and checked-in mainnet failure shapes.

tayy-i

Verdict: REQUEST_CHANGES — rebase + rewrite of two checks against current authority; defer from the v9 deploy image

Adversarial review of head 99d9607 against current main (27dbfab). Two-model panel (fable in-harness + Gemini 3.1 Pro High inline) reconciled against hand-verified code reads, plus a supplemental security pass over the SQL/connection surface.

This is explicitly not SUPERSEDED-close-it: only one of the seventeen checks (stored_lineage_backfill_coverage) is superseded by the durable coverage-frontier machinery from #151. The other sixteen — projection drains, replay markers, deferred indexes, watch-set observation, manifest/discovery presence, lineage contiguity, dataset non-empty — are performed by nothing else in the tree, and the /v1/status category error this PR fixes is real and demonstrated (July Sepolia incident). The gate is worth landing. It is not landable, or even buildable-for-v9, as-is.

Verified correct (do not re-litigate)

Genuinely read-only. connect_read_only sets default_transaction_read_only=on per connection (apps/worker/src/inspect.rs:157-172 @ 99d9607); the inspect command builds its own pool and every loader uses it; full sweep of the production diff surface found zero INSERT/UPDATE/DELETE/DDL/TRUNCATE/LOCK/pg_advisory/SET TRANSACTION outside #[cfg(test)]; no advisory locks, so it cannot block writers at the lock level; no migrations in the diff.
No SQL injection surface. All dynamic SQL is assembled from compile-time constants: count_table takes &'static str with two literal callers (crates/storage/src/data_completeness/content.rs:205-210); load_manifest_targets formats a const CTE; projection_queries whitelist-matches projection names and bail!s otherwise (.../projection_content.rs:47-154). Every runtime value is a bind. The database URL is never echoed in errors or output.
projection_apply_drained exact-equality criterion is still sound on current main. The complete-prefix capture work (20260716120000 + 20260716120100) takes LOCK TABLE ... IN SHARE MODE before reading MAX(change_id), so a visible MAX is a complete prefix; main's own blocking-work test is still cursor < MAX (apps/worker/src/projection_apply.rs:97-137 @ main). Both migrations rewind the cursor to 0, and a gate run mid-window correctly fails.
projection_replay_complete bootstrap-target parity. The PR's required-target SQL (content.rs:282-307) exactly mirrors main's bootstrap target (raw-fact cursor targets vs GREATEST(canonical, safe, finalized) checkpoint max, same -1 sentinel handling; apps/worker/src/automatic_projection_replay.rs:437-501 @ main). Marker schema unchanged; ALL_CURRENT_PROJECTION_ORDER identical on both refs.
CI is green on head 99d9607 (all nine checks) — but see mergeability below.
Chunked gap reporting and MAX_REPORTED_UNCOVERED_TUPLES output bounding in backfill_coverage.rs are correct as written.

Blockers

P1-1. DIRTY/CONFLICTING against main — and the conflicts are the load-bearing machinery. 15 conflict files. The PR moves the stored-lineage coverage reader and topic-drift guard into crates/storage/src/backfill_jobs/; main has since rewritten that exact surface: #153 added coverage/companions.rs, #151 added coverage/frontier.rs + coverage/lineage_forks.rs + migration 20260716122000_stored_lineage_coverage_frontiers.sql and moved topic drift (modify/delete conflict). This is a semantic rebase, not a mechanical one.

P1-2. Stale replay-version pin — wrong verdicts for the v9 cutover. Branch apps/worker/src/replay/progress.rs:9 pins CURRENT_PROJECTION_REPLAY_VERSION = 6; main is at 9 (:19 @ main). A gate binary built from this branch demands v6 markers (strict equality, evaluate.rs:453): it false-FAILs a correctly v9-replayed candidate and false-PASSes stale v6-era data. Version-relative design heals on rebase; this branch's binary must not ship in the v9 image.

P1-3. Deployment-profile inference diverged — permanent false-FAIL on current Sepolia. crates/storage/src/data_completeness/manifest_targets.rs:100-109 requires every active Sepolia manifest's deployment_epoch to end _sepolia_dev. #151 activated the post-audit corpus (deployment_epoch = "ens_v2_sepolia_post_audit", manifests/sepolia @ main) and main's own inference dropped the epoch requirement (apps/indexer/src/main/reconciliation/replay/profile_scope.rs:325-358 @ main). Against a current-main Sepolia database the gate resolves no profile, fails closed, and every normalization check false-FAILs forever. The PR's "replay admission and inspection share one authority" invariant is already broken by main.

P1-4. stored_lineage_backfill_coverage is superseded by and divergent from the frontier authority (#151). It re-implements the pre-#151 anti-join (crates/manifests/src/lib/views/watched/coverage.rs @ head), diverging from main's current promotion reader in both failure directions (Gemini lens concurs, high confidence, each hand-verified):

Single-row containment vs main's "gap-free union of address- and family-scoped facts whose parent job is completed": with 8192/16384/50000-block job windows and a 131,072-block verification chunk, a long-lived tuple's required interval cannot fit one fact row — systematic false-FAIL at Base scale.
No completed-parent-job requirement — a false-PASS direction main explicitly closed.
cia.deactivated_at IS NULL / de.deactivated_at IS NULL erase historical requirements main deliberately retains — false-PASS.
No discovery-admission-epoch fence (main reads required tuples in-transaction under the fence).
Hardcoded resolver family mapping vs main's shared, position-aware watched-interval CTE (#151 made same-block ENSv2 edge selection position-aware) — the required set can diverge from what intake actually watched, in both directions.
It ignores the durable stored_lineage_coverage_frontiers proof and re-scans the entire retained span every invocation (~350 anti-join chunks at Base) — the precise recomputation #151 eliminated. The correct current-world form is cheap: validate the frontier header (verified range vs retained lineage, requirement digest, admission epoch).

P1-5. watch_set_code_observation_coverage — the flagship check — false-FAILs at Base scale by construction. On current main, backfill writes raw_code_hashes only for addresses that emit a selected log (selected_seed_log_addresses, apps/indexer/src/main/backfill/fetching/materialization.rs:69-80 @ main; call sites fetching.rs:281-289), so a backfill-admitted target that never emits — the dominant class among Base's millions of discovery-admitted resolvers/subregistries — never gets an observation from backfill. The claimed remediation, the live tailer's missing-baseline pass, is skipped entirely whenever the cycle resolves via stored-lineage promotion (apps/indexer/src/main/reconciliation/persistence/code_hashes.rs:29-31 @ main — early return on StoredLineagePromoted, i.e. throughout the Base resync), and otherwise requires a 3.84M-address anti-join plus eth_getCode per missing address in one cycle — operationally unreachable. The check needs an intake-side contract change, scoping (e.g. manifest-declared targets only, discovery targets by different evidence), or an explicit documented dependency on the repair raw-code-hashes tool. (It also recontextualizes the PR's own Sepolia 352/355 diagnosis: structurally at least partly "intake never owed this row", not purely missing data.)

P1-6. normalization_caught_up_to_raw_head ignores the #151 staging input-revision fence — a durable false-PASS. Migration 20260714120000_raw_log_staging_input_revisions.sql @ main introduced raw_log_staging_input_revisions / raw_log_staging_block_revisions (+ backfill_jobs.raw_log_retention_generation), and main's own completion validators now require staging-boundary acceptance (retained-history proof: generation/epoch/through-block) before trusting next > target (adapter_sync/backlog/cursor.rs:51-128 @ main; normalized_replay_catchup/cursors.rs configured_cursor_complete @ main). The gate reads none of these tables (crates/storage/src/data_completeness.rs:455-473), so a quiesced database whose staging re-staged older blocks (revision bump below the cursor — invisible to the canonical raw-log head MAX) reads as caught up even though main's writer would rewind on its next pass. This is the one durable false-PASS the panel found, and the worst class of defect for a cutover gate.

P1-7. #168-class client materialization at Base scale. apps/worker/src/inspect/data_completeness.rs:31 materializes the full watch view (load_watched_contracts); content.rs:9-42 materializes every observed (chain, address); the evaluator builds BTree structures over both (evaluate.rs:51-126). Base watched surface: 3.84M addresses / 15.2M discovery edges (#153) — the exact shape that blew a 40GiB cgroup in #168 (fixed by #173's streaming). Fine on Sepolia; not runnable against the Base candidate, which is the gate's stated purpose. Needs the #153 treatment: server-side anti-joins/LATERAL probes, per-chain chunking, violations-only results. The runbook has no scale or runtime caveats at all.

Non-blocking (P2)

P2-1. Hours-class unbounded scans, no statement_timeout/lock_timeout. load_active_manifest_event_sources three-way joins every serving-canonical normalized_events row (56M+ at Base) to chain_lineage and raw_logs (content.rs:127-174); duplicate-height and disconnected-parent CTEs self-join retained chain_lineage (data_completeness.rs:356-381). Server-side (no OOM), but a session pinning a snapshot for hours holds back vacuum and stalls CREATE INDEX CONCURRENTLY — it can block the replay finalize's deferred index rebuild despite being read-only. Security-lens classification: resource exhaustion by operation (requires DB credentials; no unauthenticated surface). Bound it and document expected runtime and when it is safe to run.
P2-2. Frontier-lag check semantics. The window is asymmetric [-1024, +8] (report.rs:170-177; MAX_LIVE_CONTIGUOUS_GAP_FILL_BLOCKS = 1024, a hand-copied duplicate of the writer constant in reconciliation/canonical.rs:61 @ main) — the PR description's "±max" wording is a doc-code mismatch. Mid-promotion of deep stored lineage the check FAILs for the whole (potentially days-long, cf. #135) window with no advisory distinguishing "healthy, progressing" from "stuck". Not a false-PASS; needs the constant derived/cross-asserted and an advisory.
P2-3. Sparse hash-pinned backfill lineage density vs the contiguity denominator. Sparse backfill writes chain_lineage only for resolved log-bearing blocks (backfill/fetching/sparse.rs:255,289,429 @ main), while reconciliation_lineage_contiguous demands COUNT(DISTINCT block_number) == head - floor + 1 (data_completeness.rs:17-20,321-329). If any Base history was admitted via the sparse path without dense lineage, the check false-FAILs regardless of data completeness. Author should state the intended lineage-density contract for Base before this check gates anything.
P2-4. Hand-copied policy duplicates. The closure-family latch list (crates/adapters/src/normalized_replay_policy.rs:7-19 @ head) currently matches main's structurally derived set (replay/classification.rs:538-548 @ main, 11 families) but drifts silently; same for the 1024 constant (P2-2). Derive or cross-assert in tests.
P2-5. Thread hygiene. All 53 codex threads are author-replied but none reviewer-verified/resolved, and the final fix round (99d9607) has no codex re-review. The reply→verify→resolve lifecycle needs completing on the rebased head.

Notes (P3)

active_event_lineage_retained's exclusion of NULL-block-hash manifest-sync/alert events (written serving-canonical with block_hash: None by manifest_normalized_events/builders.rs:80-94 @ main) is coincidental kind-set filtering, not an invariant: declaring any non-log-anchored kind in an ABI/adapter list flips a healthy DB to permanent FAIL. Add an anchoring invariant test.
projection_replay_complete's target-coverage condition is stricter than main's bootstrap handoff; verdict-neutral (apply-drained fails too) but can misattribute an apply-lag failure to the replay check.

Panel status

Gemini 3.1 Pro (High) inline pass: completed; its six verdicts are folded into P1-4 (each hand-verified against the SQL before adoption). Fable in-harness pass: completed; its Q1/Q2/Q7 findings are P1-5, P1-6, P2-2/P2-3 (load-bearing citations hand-verified: code_hashes.rs:29-31, materialization.rs:69-80, migration 20260714120000, report.rs:170-177). Supplemental security pass: no injection, no writes/locks, no credential leakage; resource-exhaustion angle captured in P1-7/P2-1.

Recommendation for the v9 deploy

Defer — do not include in the v9 image. P1-2 alone makes a branch-built gate's verdicts wrong for that deploy, and the branch cannot merge (P1-1). The durable frontier + existing promotion invariants carry the backfill-coverage half for the cutover. If a pre-cutover completeness signal is wanted, the fastest sound path after rebase is a minimal subset — frontier-header validation, projection drains + staging-fence check, replay markers at v9, deferred indexes, dataset non-empty — with the full gate (rewritten coverage + rescoped code-observation checks) following after the deploy.

tayy-i

Supplement to my REQUEST_CHANGES review (second fable pass completed after posting; all three additions hand-verified against origin/main / head 99d9607 before adoption). Verdict unchanged.

Sharpens P1-5 (code-observation coverage). The missing-baseline pass cannot heal a stale observation, only a missing one: load_missing_raw_code_baseline_addresses selects addresses with zero non-orphaned rows and has no block-number qualifier (apps/indexer/src/main/reconciliation/persistence/code_hashes.rs:266-276 @ main), while the evaluator requires an observation at/after the latest finite start when entries share an address (evaluate.rs:130-154 @ head). A re-admitted address whose only observation predates its latest start is therefore a permanent FAIL that no automatic writer can clear — only a fresh emission or the manual repair raw-code-hashes command. Writer enumeration is complete across all three backfill paths (fetching.rs:283,334, fetching/historical.rs:170,211, fetching/sparse.rs:326-330,444 @ main): all gate on selected_seed_log_addresses, i.e. emitters only.

Sharpens P2-2 (frontier-lag window). The −1024 lineage-ahead allowance is calibrated to one promotion batch, but hash-pinned backfill stamps every revalidated block at or below the provider-finalized head as Finalized at intake (backfill/fetching/canonicality.rs:195-216 @ main), so retained lineage head jumps to the top of the backfilled span while the checkpoint advances ≤1024 blocks per live iteration. lag ≪ −1024 is the routine, healthy state for the entire Base promotion campaign: reconciliation_frontier_at_head is a quiescence-only check and should be documented (or defaulted) as such.

New P3 (latch handoff leniency). For latched chains, main's writer demands the backlog cursor's range_start == raw_fact_target + 1 exactly (adapter_sync/backlog/cursor.rs:90-101 @ main); the evaluator flags only range_start > target + 1 (evaluate.rs:369-388 @ head), so a stale backlog cursor starting below the handoff boundary can false-PASS. Low likelihood; tighten to exact equality on rebase. Related drift evidence: ENS_V2_PERMISSIONS_SOURCE_FAMILIES already changed on main after the branch point (union of latching families unchanged today, but the hand-copied policy list in crates/adapters/src/normalized_replay_policy.rs has no cross-assertion test).

Also confirmed by the second pass (already in the review as verified-correct, now with stronger evidence): projection_apply_drained exact-equality remains sound on main including the complete-prefix capture and the post-rederive cursor-ahead case (a correct FAIL mid-surgery), and active_event_lineage_retained has no false-FAIL class in code — main never deletes chain_lineage outside tests, and #24/#27 repair events carry real stored block identities.

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

LeonmanRolls mentioned this pull request Jul 10, 2026

docs: add a blue-green cutover runbook and the re-derivation checklist #143

Open

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

LeonmanRolls added 2 commits July 14, 2026 08:16

Merge PR 139 synthesis and close Wave 5 gaps

09f60c9

chatgpt-codex-connector Bot reviewed Jul 14, 2026

View reviewed changes

fix: close remaining data-completeness review gaps

d08b166

chatgpt-codex-connector Bot reviewed Jul 14, 2026

View reviewed changes

fix: align data-completeness gate with live writers

547dc2c

chatgpt-codex-connector Bot reviewed Jul 14, 2026

View reviewed changes

LeonmanRolls added 2 commits July 14, 2026 18:59

fix: align completeness checks with retained authority

8313d65

Merge remote-tracking branch 'upstream/main' into ab/wave4-sol

2dfd68a

chatgpt-codex-connector Bot reviewed Jul 14, 2026

View reviewed changes

fix: align completeness gate with promotion authority

92087f7

chatgpt-codex-connector Bot reviewed Jul 14, 2026

View reviewed changes

LeonmanRolls added 2 commits July 14, 2026 20:52

fix: close completeness authority gaps

b8f128f

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

LeonmanRolls mentioned this pull request Jul 15, 2026

fix: clear normalized replay cursor failures on successful idle iterations #152

Open

tayy-i requested changes Jul 20, 2026

View reviewed changes

tayy-i reviewed Jul 20, 2026

View reviewed changes

tayy-i mentioned this pull request Jul 21, 2026

handoff: merge priorities and open items for the server review pipeline (2026-07-17) #174

Closed

		pub(super) fn watch_set_observed(&self) -> CheckStatus {
		CheckStatus::from_pass(self.unobserved_targets.is_empty())

		.head_lag_blocks
		.is_some_and(\|lag\| lag <= self.max_head_lag_blocks)

		pub(super) fn content_present(&self) -> CheckStatus {
		CheckStatus::from_pass(self.normalized_event_count > 0 && self.name_current_count > 0)

		let projection_apply_cursor_missing =
		read.max_projection_change_id.is_some() && read.projection_apply_cursors.is_empty();

		let completed = cursor.last_completed_block_number.unwrap_or(-1);
		(completed < head).then(\|\| CursorLag {

		SELECT DISTINCT replay_version, projection
		FROM current_projection_replay_status

		AND edge.deactivated_at IS NULL
		AND edge.active_to_block_number IS NULL

		let max_change_id = read.max_projection_change_id?;
		(cursor.last_change_id < max_change_id).then(\|\| CursorLag {

Uh oh!

Conversation

LeonmanRolls commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

The check that carries the gate

First review follow-up

Second review follow-up

Third review follow-up

Fourth review follow-up

Synthesis follow-up

Final review follow-up

Previous review follow-up

Current review follow-up

Latest review follow-up

Final authority follow-up

Shape

Scope

Tests

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

LeonmanRolls commented Jul 10, 2026 •

edited

Loading