docs: add a runbook for backfilling a running indexer#142
Conversation
Closing a raw-fact coverage hole on a serving deployment is a recurring operation, and dropping the database is not a general answer: a cold start reproduces the hole, because a backfill job freezes its selected target set at creation while discovery admits targets from normalized events produced by that same job. Records the procedure, the two hard preconditions, the iterate-until-fixpoint loop, and the per-target verification. Coverage is per target: a target with no ingested logs has a floor at its admission block, not the chain head, so an `onchain_max > ingested_max` comparison silently skips exactly the targets that were never watched. Also records the provider cost model. Hash-pinned backfill resolves one block hash per block, so provider cost scales with the block range and not with the number of selected targets; batch size removes latency stalls up to the throughput ceiling but does not change the total.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 457670d74a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| -- Per-target ingested frontier. Targets with no rows come back NULL and must be | ||
| -- scanned from their admission block, not skipped. | ||
| SELECT cia.address, | ||
| MAX(rl.block_number) AS ingested_max_block |
There was a problem hiding this comment.
When a newly admitted target has any later log ingested by the live tailer, this MAX becomes that recent live block, and the next step only checks the provider above it. That misses the historical window this runbook is meant to repair (from the target's active/admitted start up to the first live-tail log), so operators can stop after a "clean" check while old raw facts are still absent. The coverage check needs to scan from each target's active range start or otherwise prove there are no gaps below the current max.
Useful? React with 👍 / 👎.
| LEFT JOIN raw_logs rl | ||
| ON LOWER(rl.emitting_address) = cia.address |
There was a problem hiding this comment.
For any serving database with more than one watched chain, this joins and groups by bare address, so a log from the same address on another chain can produce ingested_max_block and mask a missing backfill on the chain passed to bigname-indexer backfill. The verification query should filter cia.chain_id for the job's chain and join raw_logs on that same chain_id before using the result as coverage evidence.
Useful? React with 👍 / 👎.
| 4. **No leaked backfill lease.** A `backfill_jobs` row stuck in `running` holds a | ||
| lease. Inspect `status`, `updated_at`, and `failure_reason` before adding |
There was a problem hiding this comment.
In a deployment with an interrupted worker, the lease lives on backfill_ranges (lease_token, lease_owner, and lease_expires_at), not on backfill_jobs; a job in running may have an expired or cleared range lease, and a stuck range lease is what blocks reservation. This precondition sends operators to inspect only job-level status, updated_at, and failure_reason, which can miss the actual leaked lease or make them avoid a safely reclaimable job.
Useful? React with 👍 / 👎.
|
|
||
| ```sh | ||
| bigname-indexer backfill \ | ||
| --chain ethereum-sepolia \ |
There was a problem hiding this comment.
When this example is copied for an ENSv2 Sepolia deployment from a normal operator shell, bigname-indexer backfill defaults --manifests-root to manifests/mainnet, so --chain ethereum-sepolia resolves against the wrong profile unless BIGNAME_INDEXER_MANIFESTS_ROOT=manifests/sepolia happens to be set. Include the profile flag (or make the chain/profile placeholders match) so the runbook command actually selects the serving Sepolia watch plan.
Useful? React with 👍 / 👎.
| LEFT JOIN raw_logs rl | ||
| ON LOWER(rl.emitting_address) = cia.address | ||
| AND rl.canonicality_state <> 'orphaned'::canonicality_state | ||
| WHERE cia.deactivated_at IS NULL |
There was a problem hiding this comment.
For databases that retain active contract_instance_addresses rows not currently selected by an active manifest or discovery edge, this query treats every active address row as a watched target even though the backfill selector only uses load_watched_contracts/the watched plan. That can send operators to scan and backfill unrelated active instance addresses while the runbook claims the table is per active watched target; the verification query should derive the same active watched set used by whole_active_watched_chain.
Useful? React with 👍 / 👎.
| **3. Let normalization catch up.** Discovery edges are written from normalized | ||
| events, so new targets only appear after replay drains. Watch | ||
| `normalized_replay_cursors`: `next_block_number > target_block_number` and | ||
| `last_failure_at IS NULL`. |
There was a problem hiding this comment.
With the command shown above, --hash-pinned-adapter-sync defaults to auto, and manual hash-pinned backfill maps that to inline adapter sync rather than raw-only replay; inline adapter sync writes normalized events directly and does not advance normalized_replay_cursors. On that default path this wait condition can remain stale or absent even after discovery has caught up, so the runbook should either switch the command to raw-only before using cursor progress or tell operators what to check for the inline path.
Useful? React with 👍 / 👎.
| For source families with a topic-signature scan the circularity does not apply, | ||
| because ingest identity is the topic plan rather than a moving address list. The | ||
| ENSv1 generic resolver scan (`apps/indexer/src/main/ens_v1_resolver.rs:4`) and | ||
| the Basenames registry scan ([`../chain-intake.md`](../chain-intake.md) § Backfill | ||
| contract) already work this way. |
There was a problem hiding this comment.
The procedure's command uses hash-pinned backfill unless the operator explicitly chooses Coinbase SQL, but the hash-pinned topic-first path only scans all emitters for the ENSv1 generic resolver; the Basenames registry topic-signature scan is the Coinbase SQL source-family special case. If this exception is read as applying to a hash-pinned whole-chain Base run, operators can treat one pass as non-circular even though newly discovered registry emitters are still absent from that pass's address filter.
Useful? React with 👍 / 👎.
The verification universe now mirrors the backfill selector (manifest-declared plus admitted discovery instances, chain-scoped) instead of every active contract_instance_addresses row, and coverage is proven from each target's active-range start with bounded provider windows rather than trusted from the stored maximum log block. The lease precondition inspects backfill_ranges lease fields (the lease does not live on backfill_jobs). The example pins --manifests-root and --hash-pinned-adapter-sync inline; step 3 waits on job range completion because inline sync writes normalized events and discovery edges directly and never advances normalized_replay_cursors. The topic-scan circularity exception is scoped: address-free Basenames registry scanning is the Coinbase SQL source-family path, so a hash-pinned whole-chain Base run still requires iterative passes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd922b2f4e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| [Verification](#verification) and compare `(source_family, | ||
| contract_instance_id, address, scan_from_block, scan_to_block)`. If the set |
There was a problem hiding this comment.
If the finalized head advances between passes, including scan_to_block in this tuple makes every open-ended active target look different even when discovery admitted nothing new. In that normal running-indexer scenario, following this step can trigger another full backfill only because time passed; keep the same verification_head for the before/after comparison, or compare target identity and scan_from_block separately and use scan_to_block only for the provider coverage scan.
Useful? React with 👍 / 👎.
| --chain ethereum-sepolia \ | ||
| --from-block <lowest admitted start across active targets> \ | ||
| --to-block <finalized head> \ | ||
| --hash-pinned-adapter-sync inline \ |
There was a problem hiding this comment.
With the current hash-pinned runner, the activity-scoped code-observation path is only taken for RawOnly sparse materialization; the inline path still builds code-observation requests from all selected addresses active at each block. For a whole-active chain over a large target set—the exact case called out in precondition 1—copying this command can still write O(selected targets x blocks) raw_code_hashes and exhaust the volume before discovery converges, so the runbook needs either a raw-only-plus-replay path or a code version where inline mode is fixed before prescribing inline.
Useful? React with 👍 / 👎.
The "why one pass is not enough" section stated the Basenames registry family was address-free only on the Coinbase SQL source-family path, and that a hash-pinned Base job was always address-filtered. The hash-pinned path now also scans-all for a `--source-family basenames_base_registry` job, over the registry manifest ABI event topics with no address filter and a target-set-independent job identity, so mid-job discovery expansion cannot invalidate it. State the exception as keyed to the source-family selector rather than to a provider: both ENSv1 generic resolver and the Basenames registry family scan-all on a source-family job, so those passes are single-pass; a `whole_active_watched_chain` job stays address-filtered and circular and still iterates. Note the deployed-image requirement for the new behavior.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3413e283af
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| SELECT COUNT(*) AS rows, | ||
| COUNT(DISTINCT code_hash) AS distinct_code_hashes | ||
| FROM raw_code_hashes | ||
| WHERE chain_id = 'ethereum-sepolia'; |
There was a problem hiding this comment.
When this runbook is used for any non-Sepolia repair (mainnet/Base are explicitly in scope), this final check always reads ethereum-sepolia, so it can report stable counts even while the job's actual chain is still writing per-block raw_code_hashes. That undermines the policy verification; use the same :'job_chain' variable as the coverage query.
Useful? React with 👍 / 👎.
| (`apps/indexer/src/main/backfill/fetching/log_ranges.rs`). | ||
| - The Basenames registry family (`basenames_base_registry`) scans-all on a | ||
| source-family job — `--source-family basenames_base_registry` — on both the | ||
| Coinbase SQL path and, since #149, the hash-pinned path. The hash-pinned |
There was a problem hiding this comment.
When an operator follows the default hash-pinned path for --source-family basenames_base_registry, this one-pass claim is not implemented: the hash-pinned topic-first selector in apps/indexer/src/main/backfill/fetching/log_ranges.rs only returns a scan-all topic plan for the ENSv1 generic resolver, while Basenames registry scan-all is gated in the Coinbase SQL planner/source-identity path. Fresh evidence: the current tree has no reservation_execution/scan_all.rs and coinbase_sql_uses_basenames_registry_scan_all is the only Basenames scan-all gate, so a hash-pinned Basenames repair can still freeze the discovered address set and miss recursively admitted registry emitters.
Useful? React with 👍 / 👎.
| **4. Check for newly admitted targets.** Rerun the active watched-range query in | ||
| [Verification](#verification) and compare `(source_family, | ||
| contract_instance_id, address, scan_from_block, scan_to_block)`. If the set | ||
| grew, return to step 2 with a fresh idempotency key. The new pass covers the |
There was a problem hiding this comment.
When a historical pass discovers an earlier active edge for a contract instance that was already in the watched set from live intake, reconciliation can replace the tuple with a lower scan_from_block without increasing the set size. The previous pass selected that target only from the later start, so treating only growth as the loop condition can stop while the newly widened historical slice is still missing; rerun on any tuple change, especially a lower scan_from_block.
Useful? React with 👍 / 👎.
| For each returned target range, issue bounded `eth_getLogs` windows beginning | ||
| at `scan_from_block`, filtered to that address, and compare every returned log | ||
| identity with `raw_logs` on the same chain. For one provider result, the storage |
There was a problem hiding this comment.
For ENSv1 generic resolver topic scans, and for any Basenames registry/source-family scan-all path, the job selects only the declared event topics plus same-transaction context; it does not promise to store every unrelated log emitted by those addresses. An address-only eth_getLogs comparison will therefore mark a clean topic-scanned pass as missing whenever the emitter has other events, so the provider verification needs to use the same topic/source-family filter for those paths.
Useful? React with 👍 / 👎.
Server-pipeline adversarial review (head 3413e28 vs main 27dbfab) — REQUEST_CHANGESNot superseded (main has no competing runbook; the iterate-until-fixpoint core and per-target verification SQL are sound — the big CTE was verified schema-faithful against the baseline migration and the watched-interval predicates). But the runbook predates 16 commits of machinery and needs a real revision: P1 — the default job shape is the documented wedge. Step 2's "omit a selector to take whole_active_watched_chain" with hash-pinned + inline sync is exactly the #137 shape that stalled CPU-bound at Base scale (3.84M targets, job 719, zero fetch progress); the ratified shapes are per-family jobs / coinbase-sql, neither mentioned. A verbatim operator on Base wedges with no diagnostic. Selected P2s (full list in the session record): the cost model understates provider quota ≥3× in the mode the runbook pins (per-block getBlockByNumber + full-tx getBlockByHash + getBlockReceipts, verified in fetching.rs/block_transaction.rs); the scan-all keying rule is false for ENSv1 (whole-chain jobs DO topic-scan ens_v1_resolver_l1 — source_scope.rs:145-152); precondition 2 ("live coverage self-corrects") is unsatisfiable on a main-built image while #141 is open; the step-2 command's missing --database-url silently falls back to a default-credential localhost DB (storage lib.rs:394-399) — a security-adjacent unsafe-intermediate-state hazard; no integration with the coverage-facts/frontier machinery the jobs now feed (#125/#138/20260716122000, and #154's overclaim caveat); the Basenames source-family pass silently downgrades to raw-only so the runbook's own fixpoint check reads false convergence; coinbase-sql presented as coequal despite the #149-documented warehouse omission. Codex triage NOT satisfied: latest-head 1 P1 + 5 P2, zero replies/resolutions. Two are stale (the inline code-hash scoping P1 was fixed by #133's selected_seed_log_addresses; the no-scan-all-in-tree P2 predates #149) — the rest stand. Security lens (added per the corrected review scope): no credential exposure in commands/pasted output beyond the DB-fallback item above. Remediation routes through the standard pipeline after #141 lands (its openness invalidates precondition 2 as written). |
3 participants
Why
Closing a raw-fact coverage hole on a serving deployment is a recurring operation, and dropping the database is not a general answer — on mainnet it is not an option at all, and even on a testnet a cold start reproduces the hole.
The reason is circular: a backfill job freezes its selected target set at creation (
whole_active_watched_chainselects "every active watched target ... at job creation",docs/chain-intake.md§ Selector modes), while discovery admits targets from normalized events, which are derived from the raw facts that same job produces. A target discovered by a pass was never in that pass's own filter. So the procedure is an iterate-until-fixpoint loop, not a single command.What this records
onchain_max > ingested_maxcomparison silently skips exactly the targets that were never watched. That is the failure mode the runbook exists to correct.GET /v1/statuscannot be used for this:projection_lag_blockstracks head reconciliation, sochain_lineagecan be gapless to the chain head whileraw_logsholds nothing for a watched target.apps/indexer/src/provider/block_transaction.rs:173-192), so provider cost scales with the block range and not with the number of selected targets.BIGNAME_INDEXER_JSON_RPC_BATCH_ITEM_LIMITremoves latency stalls up to the throughput ceiling but does not change the total; the block-hash loop is sequential and does not consumeBIGNAME_INDEXER_JSON_RPC_BATCH_CONCURRENCY, which applies only to receipt fetches (apps/indexer/src/provider/transaction_receipts.rs:532).Documentation only. No code, semantics, or coverage change.
Related
Preconditions land in #133 and #141. This runbook is safe to merge before either, but the procedure should not be run until both are deployed.