fix(indexer): bound the startup discovery-adapter sync — resume-capable checkpointed pages (#218)#219
Conversation
Route startup ENSv1 discovery-derived full-source passes through distinct resumable checkpoints and keep authority page events private until identity materialization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p
|
Adversarial review gate (two lenses on 8d05751) — no P1; one P2 (an honesty correction, applied to the PR body), ops-monitoring P3s recorded, residual lanes filed as #220. Fable pass — the load-bearing boundedness claim verified with quantified constants. Subregistry: genuinely O(page) (fresh 10k-log map per page, latch-bounded index-supported scans, the #173 streamed reconcile with its SQL-COUNT coarse cap enforced before any in-memory load, 20k/5k emit paging; the one cross-page accumulator is the migrated-node guard, bounded by the 2020 eth registry migration — tens of MB). Authority: O(name-state), ~single-digit GB/chain — identical to the already-shipped replay-lane residency, no new accumulation; the O(matched-logs) event term (the OOM's dominant component) moved to DB staging. Equivalence: the latch uses the same canonicality set over the same scan population as the old pass; boot sequencing makes mid-pass writes impossible; mid-span corpus changes between boots are caught by the trigger-maintained revision journal's reset check (not just target monotonicity); the fixed-point repeat makes the startup pass at least as complete as the old one. Lifecycle: epoch fence threaded through every checkpoint writer (surgical jsonb_set in the publish loop) with reset-on-drift tests via the real Gemini pass adjudicated: cross-page accumulation — confirmed-and-quantified (the guard is tens of MB; the real residency is the authority name-state, now correctly framed); resume re-latch — refuted (fresh-boot-equivalent semantics + revision-journal reset); missing epoch in the resume context — refuted (the latched epoch lives in state_payload and survives every writer, test-proven); non-atomic flush/checkpoint — refuted (idempotent identity upserts + atomic publish pages, per the documented crash-window semantics); cleanup leak — refuted-as-designed (partial retention is the resume feature; cleanup is scope-bound with a completed-status guard). Ops watch items for the first fixed boot (recorded in #220's tail): fixed-point double scan + ~25–50GB transient checkpoint-staging growth across both chains on a 94%-full disk, then the cascade-delete/autovacuum tail — disk monitored during the roll. Deactivation caps cannot trip spuriously on the raw-only-frozen edge state (candidates ≈ 0). Merging on CI green + connector triage; then image rebuild, stopgap off, full deploy resume. |
Closes #218 — the live deploy blocker: the post-drain startup discovery sync ran the unbounded full-source ENSv1 derivation (whole-span assignment map with raw-log clones + in-memory reconcile), peaking at ~55–85GB on the Base pass and kernel-OOM-looping the indexer every ~15 minutes at post-catch-up two-chain scale. Regression vector was #141 putting the pre-existing unbounded path on every auto boot; the bounded #168/#173 machinery existed but the startup entrypoint bypassed it.
Change
startup_adapter_owned_raw_log_state, scopestartup_adapter_sync, fixed range start 0, latched target — collision-proof against replay'sfull_closurescope, and fenced bydiscovery_admission_epochs.BIGNAME_INDEXER_DISCOVERY_FULL_RECONCILE_MAX_DEACTIVATIONSnow apply), checkpoint-paged event emission; unwrapped-authority events stay checkpoint-private until identity materialization then publish in 20,000-event transactions. (Per the adversarial review: the authority path is O(name-state) — ~single-digit GB/chain, byte-identical to the already-shipped replay-lane residency, not a new accumulation — versus the ~55–85GB O(span) class this PR removes; the dominant O(matched-logs) event term is what moved to DB staging.)Verification
Equivalence: exact edge/event parity with the former full pass. Kill/resume audit: page counters
[1, 2, 3]— no rescan from zero. Page-bounded staging-row assertions; cleanup, higher-target resume, authority-drift, empty-source, event-visibility coverage. Adapters 304 / indexer 514 / manifests 109 / CI-equivalent e2e 90 — all passed; fmt/clippy/file-size clean (two allowlisted files shrank). Codex$verify-loop: definitive CLEAN.Deploy note: the live indexer currently runs the documented
raw-onlystopgap; this fix rides the next image, after which the stopgap comes off and full adapter derivation + replay catch-up resume.🤖 Generated with Claude Code
https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p