[matthewdarwin]

When a live block can't be linked, ForkableHub.linkLiveUsingOneBlocks walks the one-block store from the LIB to the end of the store, and decodes every unknown file into the forkable. One-block files above the live block can't help link it. This change makes the walk stop at the live block's number, through a new WalkOneBlocksStoreRange.

How it was found

Setup: reader-node, merger, firehose and substreams tier1/tier2 run in one firecore process on a single node, with local file:// stores. The hub is fed straight from that reader's gRPC: there is no separate relayer.

The chain has large blocks, up to about 190 MB decoded each.

• A crash lost one block: the reader had printed it, but it was never written as a one-block file.
• The merger stopped at the gap. Above it, several thousand one-block files piled up that it can't merge or prune, and later crashes left holes near the top of the pile.
• From then on, every restart went the same way:
  1. The bootstrap failed with "most recent one block is not linkable".
  2. The hub started without a LIB.
  3. The first live block from the reader, which was re-processing from a point below the gap, triggered linkLiveUsingOneBlocks from its own LIB to the end of the store.

Result: the hub decoded the whole pile, and the process was OOM-killed (128G) within about 90 seconds of every start.

The gap itself shouldn't happen. A follow-up PR will improve how reader cursors work, so a block that was printed but never stored is no longer skipped on restart. This PR is about the hub's behaviour once such a gap exists.

Behaviour change

The lookup still reads the live block's LIB and everything up to the live block, so two things are unchanged:

• the reconnection check (blockFromFile.Number == blk.LibNum);
• the filtering of copies of the same block written by several readers (unknownOneBlockFiles).

Files above the live block are left alone, and the hub links them later as they arrive live.

Reproduction

firecore from develop, built twice, with this branch's base and with this branch. The same single-node setup, reader-node merger firehose, with a synthetic chain:

• 4 MiB blocks, each block its own LIB;
• merged blocks to 999, then one-block files from 1,300 to 4,000 above a gap, minus a few near the top so the bootstrap fails;
• the reader then restarted from 1,101 at 20 blocks/s.

	before	this branch
one-block lookup for the first live block	5.5 s, every file above it decoded	6 ms, files up to the live block only
hub ready	142 s later, only once the reader passed the top of the stranded files. Until then every live block was dropped as below the forkable's LIB	21 ms after the lookup
peak RSS	3.6–4.0 GB	2.5 GB

Without the bound, the decoded files also move the forkable's LIB to the top of the stranded range. Live blocks below the LIB are dropped (#70), so the hub serves nothing until the reader catches up, while holding all of it in memory.

On the affected node

firecore v1.21.1 with only this change (its bstream, 775b59f, is this branch's base). The bootstrap still failed as before. The lookup for the first live block took 4 ms, and the hub was ready 31 ms later. There were no OOMs afterwards, and the heap settled at 2–5 GB once the bootstrap's blocks were released.

Not tested

A distributed setup, with readers, relayers, merger and firehose on separate nodes. The change only bounds the walk's upper end, so no difference is expected there. But a relayer-fed hub reaches this path in its own ways (reconnects, #79), and those weren't exercised.

Tests

• TestForkableHub_LinkLiveUsingOneBlocks_StopsAtLiveBlock: a hub without LIB, one-block files below and above a gap, and a live block below the gap. It fails on develop, which decodes the files above, and passes here.
• TestForkableHub_WalkOneBlocksStoreRange.
• go test ./... passes.

[sduchesneau]

good catch