fix(indexer): make automatic replay two-phase — catch-up output ≡ live ingestion (#157)

[tayy-i]

Refs #157 (core divergence fix; tracker stays open for the remaining acceptance items — see the issue comment), implementing the ratified option (a): automatic catch-up for closure-adapter chains now runs two phases, closing the known divergence where catch-up omitted the stateless label-preimage pass.

Lifecycle

1. Phase 1 — stateless block-derived pass over the entire latched catch-up span, physically paged by the configured raw-log cap with whole-block boundaries preserved. Uses the same canonical raw logs, admitted manifest ABI/source scope, decoder, event identity, provenance, canonicality, and normalized-event upsert path as live ingestion.
2. Phase 2 — the existing checkpointed stateful/dependency full-closure pass, unchanged semantics.

Both phases run under the same deployment-profile/chain advisory ownership lock; the raw-log input version is captured before phase 1 and accepted only after phase 2; only then does the shared raw_fact_normalized_events cursor advance. Consequences: normalized_replay_cursors_complete now means both phases completed (the worker cannot begin projection replay between phases); the #141 backlog→plan-reload→latch handoff is unchanged and cannot release live polling early; a crash after phase 1 leaves the cursor pending — restart repeats the identity-idempotent stateless upserts and resumes the checkpointed closure pass. Manual replay semantics unchanged.

Acceptance gate — flipped and strengthened

CATCHUP_EQUIVALENCE_CONTRACT flips MissingStatelessLabelPreimages → Full (the #193/#198 scenario now demands full corpus equality), and per the #198 review requirement the Full arm additionally authenticates the receipt-reconstructed PreimageObserved row byte-for-byte in both corpora — preserving the replay-path authentication canary post-flip.

Verification

fmt (+e2e fmt), clippy, file-size (875 files, no allowlist), sync-refs (11 refs); indexer 509 passed / worker 258 / CI-equivalent e2e 83 passed with the Full contract. Codex $verify-loop round 1 found an unpaged stateless span (reproduced with a focused failing test, fixed, retested); round 2: no concrete findings.

Deploy note

Merging does not touch the live stack (the in-flight recovery runs the current single-phase image to completion). On the next deliberate image rollout, an in-progress catch-up with a pending cursor resumes into the two-phase lifecycle — phase 1 is idempotent over already-derived rows.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db284879a5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Hold the raw-log fence before phase-one writes

When a raw-log staging revision for a consumed block commits while automatic two-phase catch-up is running the new prelude, prelude().await has already durably upserted stateless normalized_events, and the later accept_newer_revisions_after only aborts cursor publication. On retry, the new canonical selection may no longer include the old raw-log identity, so those already-committed stateless rows can remain canonical even though the raw input was rejected; this also contradicts the new storage contract that the raw-log input-version fence remains session-wide (docs/storage.md:868).

Useful? React with 👍 / 👎.

[tayy-i]

Adversarial review gate, round 1 (two lenses on db28487) — no P1/P2 correctness findings; the core semantics are proven sound. Remediation commit for three operational P3s in flight; PR linkage corrected to "Refs #157" per the scope finding.

Fable pass — completion fence proven airtight. The only production writer completing the raw_fact_normalized_events cursor is advance_cursor, called from exactly one site, and every route to it includes phase 1: closure chains run the prelude inside the lock session before the closure sync (any prelude error aborts the session); non-closure chains run all in-scope stateless adapters inline (inherent phase 1, not a no-op, not a hang); the pending-Base-rederive empty-adapter branch still runs the prelude with clean empty-range termination; ops-catchup and manual replay never touch the cursor; and the old single-phase entry point is now #[cfg(test)]-only — unreachable in production. Input-version TOCTOU is a three-layer fail-closed fence (in-session capture→accept, publication re-accepts with the earlier rewind-inspection version, stateful checkpoints self-reset on consumed-span change). Crash-resume verified by test (cursor stays pending at the phase boundary; retry identity-idempotent, exactly one PreimageObserved). Phase-1/phase-2 adapter sets are provably disjoint — no double-derivation. Paging is keyset over the canonical-replay partial index (no OFFSET): at Base's 56M-row scale ≈560 pages, ~45–90 min per full phase-1 pass. Manual replay byte-for-byte unchanged (75/75 replay + 19/19 catch-up suites green locally). The e2e Full arm is strictly stronger and the two-phase path is genuinely exercised.

Gemini pass — adjudicated: its orphaned-phase-1-rows scenario dissolves against the identity-keyed idempotent upserts + checkpoint self-reset + canonical-only paging (same exposure and same resolution as live ingestion); its no-durable-phase-1-frontier concern is real but quantified acceptable at ~1h/pass — its operational manifestations are exactly fable's P3s below. Its paging and Full-arm analyses independently confirmed clean.

Remediation (in flight): (1) retention-authority validation moves ahead of the prelude — a misconfigured chain should fail fast, not burn a phase-1 pass per retry; (2) coverage-recovery's full-span phase-1 re-run (×32 attempt cap) gets scoped to the recovered spans if cleanly achievable, else documented with the bounded multiplier; (3) the runbook gains the single-phase→two-phase upgrade rule: cursors completed under a pre-two-phase image are Idle forever and need the manual stateless-replay stopgap, while an in-progress catch-up upgraded mid-flight resumes correctly at the cost of one phase-1 pass — deploy the two-phase image before in-flight cursors complete where feasible (this is how the live recovery will be sequenced).

Scope (P3-4): linkage changed to "Refs #157" — this PR closes the ratified core divergence; the tracker's remaining acceptance items (catch-up supervision, backlog retry budget, health states, covered_backfill_only ledger graduation, defect-3) stay open on the issue, enumerated there.

CI: the sole failure is the known targeted_rebuild_serializes_claim_publish worker flake (zero worker files in this diff); rerun queued behind the still-running shards. Merge holds for all-PASS + CLEAN per the gate.

[tayy-i]

Delta review gate (defe3ee + d10a035 + 3361e55, +1622/−194 — reviewed as its own unit since it post-dates the base review) — CLEAN at P1/P2, both lenses.

Fable delta pass: span arithmetic inclusive and gap-free through every edge shape (with a release-mode ensure! backstop behind the debug assertion); no TOCTOU on the revision fence — the acknowledged version is snapshotted before the checks, commit-order = revision-order holds via the xact-scoped advisory chain lock, and a mutation landing after the checks has a higher revision that the outer cursor acceptance catches (fail-closed to full-span replan); mutation-location coverage is total (below-span fails, in-span-outside-pending widens — including above-span-top-below-target, which cannot exist as requirements are bounded to the target); the 32-attempt bound is monotone with span memory capped; the ENSv2 proof rebuild mints nothing — same ensure_proof_through, same locks, all generation/epoch/coverage/witness checks, never lowers a valid proof, never touches the revision the acceptance fence reads; manual replay is validation-equivalent with two deliberate, tested deltas; every base-review property (single cursor-advance site, three-layer fence now strictly tighter than the pre-delta unconditional re-acknowledgment, adapter disjointness, crash degrading to full-span) re-verified. The widen test was proven load-bearing: without widening, outer acceptance alone would advance the cursor missing a concurrent in-span log's preimage — the test targets exactly that hole. 48 replay-net tests run locally, 0 failures.

Gemini delta pass: independently clean on span arithmetic, fence ordering, livelock bound, and crash safety; its one cannot-determine (does the proof rebuild enforce the bypassed path's invariants) is the question fable answered above with the shared-implementation evidence.

Dispositions: P3-1 (a foreign non-ENSv2 writer hitting a recovered below-range block mid-session escapes phase-2 re-derivation — strictly narrower exposure than the base's unconditional acknowledgment, no regression) — noted. P3-2 (the fence's two fail-closed arms lack direct tests) — closing now: test-only commit in flight adding both (below-span fail; retention-generation change), each asserting the typed failure, no cursor advance, no partial acknowledgment. P3-3 (cursor last_* stats reflect the final scoped attempt — observability) — noted for the #186-family metrics work.

Merge follows the test commit + CI + final-head connector triage.

[tayy-i]

Final-head triage: the one inline comment on 1e4f772 ("hold the raw-log fence before phase-one writes") was authored at 07:28 — before the commits that implement exactly what it asks for existed — and was re-anchored forward by the force-pushes. The current head's answer, verified by both delta lenses: 3361e55's recovery revision fence (in-span mutations widen the next phase 1 to the full span before the revision is acknowledged; below-span mutations fail the attempt for durable replan; the acknowledged version is snapshotted before the checks, so a mid-window commit is never acknowledged and the outer acceptance forces a both-phase re-run), plus the base-review property that acceptance failure retries BOTH phases against the new canonical selection with identity-keyed idempotent upserts, and lineage/repair machinery owning canonicality for raw-log identities that leave the canonical set — the same exposure and resolution live ingestion has. The storage.md contract paragraph it cites was updated by the same commits.

Attribution-fix round (1e4f772) — CLEAN, no P1/P2: equivalence with live holds by construction (single shared code path; candidate intervals and live scopes derive from the same watched_intervals view; the hard-error arm unreachable for automatic/live paths); interval semantics inclusive/open-NULL matching the SQL; priority and fail-closed verified against the full manifest corpus — including the discovery that the pre-fix branch would have wedged live ethereum ingestion on the dual-declared L1 resolver, so the fix closes a broader class than the reproduced Base wedge. Corpus-lint follow-up filed as #215 (the equal-rank bail is latently armed for one mainnet address, gated only by preimage-topic non-collision). Full gate across the PR: base review clean, delta review clean, attribution review clean, merged-union e2e 90/90 (the suite that caught the wedge), CI 10/10, connector 👍. Merging.