fix(indexer): bounded in-place retry for the raced reth head refresh (#210)

[tayy-i]

Closes #210. The live recovery deploy hit this twice in ~3 hours: the reth-db source's head refresh races reth's own tip advancement — best number resolves but the canonical-hash lookup at that height returns None — and the whole indexer process died each time, paying the full startup-scan tax and aborting all in-flight work (post-#208 this would also restart an in-flight phase-1 stateless pass from scratch).

Change

• Narrow typed retry condition: only the exact raced shape — a fresh chain_info() resolving a best number with zero best hash while block_hash(best_number) returns None — retries, with a fresh best-number+hash re-read each attempt.
• Bounds: 5 attempts, 150ms fixed backoff (max 600ms added latency).
• Fail-closed preserved: exhaustion returns the final error with the existing message unchanged (persistent absence = genuine DB problem stays loud); block-fetch errors, safe/finalized-head errors, and every other DB error keep existing behavior untouched.
• Observability: each retry logs the raced height at DEBUG; recovery logs once at INFO with attempt count.
• Hooked at the single head-refresh chokepoint (fetch_chain_heads_sync); driver/classifier logic isolated in reth_db/head.rs.

Verification

Focused reth tests 4 passed (fresh-height recovery, persistent exhaustion, unrelated-error bypass, typed-only classification — via the narrow driver seam, no new datadir stub machinery); feature-enabled provider filter 93 passed. fmt/clippy/file-size clean. Codex $verify-loop: clean.

Deploy note: intended to ride the next image rebuild alongside #208 so the Base phase-1 pass is protected from the ~2-3-hourly race crashes observed live.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C9RTnbCyhqEUSPc7xCK76p

[tayy-i]

Adversarial review gate (two lenses on e794d6e) — fable: CLEAN, zero findings at any severity; Gemini: three precise no-defects, two open items both resolved by fable with repo/upstream evidence.

Verified highlights: the retry type is private and constructed at exactly one site — byte-for-byte the old failure site — so nothing else can trigger the retry, and no previously-succeeding shape changes behavior; exhaustion returns the raw error whose Display is the identical string the old code produced (the outer wrap reproduces the exact live failure_metadata text from the #210 incident — monitors see an unchanged signal); against pinned reth v2.2.0 source, chain_info's zero best-hash IS the same absence the explicit block_hash==None re-check sees (unwrap_or_default swallowing), making the compound condition one coherent race signature rather than an over-broad OR; every attempt opens a fresh provider read-tx (no stale height spin); the backoff sleeps run on tokio's blocking pool via the sole spawn_blocking caller, with no timeout-wrapped call sites and the healthcheck path untouched — worst case +600ms in the raced state versus a full crash plus the #135 startup tax. Tests: 4/4 (with the classifier test guarding against a future string-matching regression by asserting a text-identical untyped error is NOT classified); clippy -D warnings clean; feature-gated tests confirmed to run in CI. Gemini's stale-read test-gap adjudicated acceptable: the closure wiring is one stateless line, untestable without a real factory, with the loop-level fresh-read seam-proven.

Merging on CI green + connector verdict. Deploy note: rides the next image rebuild alongside #208 to protect the Base phase-1 pass from the observed ~2–3-hourly race crashes.