Decouple finalization from head ingestion#1
Merged
Conversation
Part of NET-408
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rust port of the same fix as subsquid/squid-sdk#499
Problem
The EVM hot-block data service spiked block-lag every ~6.4 min, locked to
block_number % 32. Root cause: when Ethereum finalizes an epoch (~32 blocks at once),finalize_stream's prober confirmed finality and emitted a burst of finalized-only batches (blocks: vec![]) through the single stream consumer, serializing ahead of head ingestion and stalling the head-poll loop.Fix
Take finalization off the data-loading path (
crates/evm-source/src/source.rs,finalize_stream):probe_rxarm of theselect!now only records the confirmed head incurrent_finalized— it no longer yields a{blocks: vec![], finalized_head}batch.current_finalizedto each head batch, so the finalized head rides the next head block.The stream now yields only head batches; the finalized head is delivered on the next one. Finality lags the head by minutes, so a sub-block delivery delay costs nothing. Probe batch size stays at 5.
This also removes a metric artifact:
sqd_hotblocks_last_block_lag_mswas re-stamped on finalized-only batches with the head's stale age, producing 10–20 s gauge readings. With no finalized-only batches, the gauge is stamped only on head advances.Validation
The TypeScript port of this change was benchmarked live (Dwellir Ethereum, 3 epochs): the
% 32epoch-lock disappeared (residues 12/13 went from 5–8 s spikes to 0 blocks >5000 ms), max blockAge dropped 13550→6000 ms, and finality advanced in correct single +32 jumps. The Rust consumer appliesfinalized_headand runstrigger_update/compactper batch identically, so the behavior matches. A local Rust run against the same endpoint confirms it.🤖 Generated with Claude Code