[matthewdarwin]

This adds --reader-node-cursor: the reader node keeps the reader's cursor and gives it back to the reader when it starts. A reader then restarts from what firecore stored, not from what it printed.

This is the follow-up announced in streamingfast/bstream#88, which fixed how the live hub behaves once a block is missing. This PR stops the block from going missing.

The problem

A reader that records a block as done as soon as it prints it can lose blocks for good. Between the FIRE BLOCK line and the one-block file, the block sits in firecore's in-memory pipeline:

• the line channel (10,000 lines);
• the blocks channel (--reader-node-blocks-chan-capacity);
• the local one-block file awaiting the next upload pass (every 500 ms).

When firecore and the reader die together (a crash, an OOM kill of the container, SIGKILL), everything in that pipeline is lost, while the reader's own record says it was printed. On restart the reader continues after it. The merger then can't link past the gap ("too many consecutive unlinkable blocks", forever), and the live hub stops with "cannot link block after reconnection, restart required".

blockpoller behaves this way: fire() writes the block to stdout, then saveState writes cursor.json. On restart initState marks the last fired block as fired, and the poller fetches the next one. Pollers built on it are exposed in the same way. A stateful reader that commits its own state independently of firecore is exposed too, and can't produce the lost blocks again without restoring an older snapshot.

In a synthetic test (below), the window was up to about 15,000 blocks while catching up: at a SIGKILL the reader had printed up to block 15,462 and firecore had merged up to 699.

What this adds

The cursor file

With --reader-node-cursor, the reader node keeps reader-cursor.json in --reader-node-working-dir:

{
  "block": {"num": 4605, "id": "53b6…6403"},
  "lib_num": 4595,
  "value": {"block": {"num": 4605, "id": "53b6…6403"}, "value": "bmV4dD00NjA2"}
}

• block is the last block in the one-block store such that every block printed before it is stored too.
  • The archiver queues each block before writing its local file, in the order the reader printed them.
  • The file uploader marks each block once PushLocalFile returns. Uploads complete in any order.
  • The cursor moves over the stored part of the queue that has no gap.
• lib_num is that block's LIB.
• value is the reader's own value (FIRE CURSOR, below) for the last stored block that had one.

How it's written: atomically (tmp file, fsync, rename, fsync of the directory). One goroutine writes it, always with the latest cursor, because uploads complete faster than a file can be synced. The archiver closes the tracker after its final upload pass, which writes the last cursor.

What it isn't: it is local to the reader node, like a poller's cursor.json, and is not derived from the block stores. Several readers can work on different ranges with shared stores, and the stores can't tell whose blocks are whose. With no cursor file, the reader starts where it is configured to, as today.

Templates

New --reader-node-arguments templates are read from the cursor file each time the reader starts. The superviser takes an arguments function (SetArgumentsFunc), called before each start.

Template	Value
{cursor-file}	the cursor file's path
{resume-block-num}, {resume-block-id}	the cursor's block
{resume-lib-num}	its LIB
{resume-cursor}, {resume-cursor-block-num}	the reader's value and the block it was given for

Each is empty when there is no cursor yet: --resume-block={resume-block-num} becomes --resume-block=.

FIRE CURSOR

FIRE CURSOR <block_num> <block_id> <value>

A reader may print this after a block. firecore keeps the value with that block and writes it to the cursor file once the block is stored. The value is whatever the reader wants back on restart: a page token for an RPC poller, or a hash of a stateful reader's state, so it can check the state it rebuilt.

• Format: the value has no whitespace, so it is a single argument.
• Compatibility: an older firecore ignores the line, as it does any unknown FIRE line. A reader that never prints it gets a cursor without a value.
• Implementation: handled in ConsoleReader.readLine, the text path of consoleline.Splitter.

Enabling it before the reader supports it

--reader-node-cursor changes nothing for a reader that doesn't know about it, as long as --reader-node-arguments doesn't use the new templates:

• the reader gets the same arguments and resumes from its own state, as exposed to lost blocks as before;
• firecore keeps reader-cursor.json anyway, which already tells an operator how far firecore stored, compared with the reader's own marker;
• with no FIRE CURSOR lines, the cursor has no value, and nothing waits on the reader.

The cost is one small bookkeeping entry per block until its upload completes, plus the file rewritten as uploads complete. A failed write is logged and changes nothing else.

Two cases do change behaviour:

• Templates feeding a flag the reader doesn't have. Writing --resume-block={resume-block-num} for such a reader makes it reject the flag and not start. That shows at the first start.
• A chain whose superviser can't resolve its arguments again (one not embedding superviser.Superviser). If the arguments use a cursor template, firecore refuses to start, because the values would go stale after the first restart. Without templates, the flag is accepted and only the file is kept.

One more: if reader-cursor.json exists but can't be read, the reader doesn't start, so a bad cursor is never silently ignored. The file is replaced atomically, so it can't be left half-written.

What readers do with it

• A reader that can produce any block again (an RPC poller) restarts from {resume-lib-num} + 1. The blocks above the LIB are printed again, and the one-block store, the merger and the hub ignore them as already known.

• A reader whose state only moves forward (it indexes the chain into its own database) has two rules:

  1. It commits its state at block N only once {cursor-file} shows N or later. It reads the file while it runs.
  2. On start, if its state is past {resume-block-num}, it refuses to start rather than skip blocks, because those blocks can't be produced again without an older snapshot.

  After an ordinary crash such a reader indexes again, silently, up to the cursor, and prints from the next block.

Follow-up: pollers

Readers have to opt in, so this PR alone protects no existing chain. blockpoller is the place to start, because most RPC readers are built on it (firehose-solana, firehose-ethereum's RPC poller, firehose-bitcoin, and the Starknet, Stellar, Tron and Injective readers, among others).

A follow-up PR would change blockpoller to:

• Take a resume block from its arguments, e.g. fetch rpc {first-streamable-block} --resume-lib={resume-lib-num}. When it's set, the poller starts at the LIB + 1 and ignores cursor.json, as IgnoreCursor() does now. The fork state the poller keeps in cursor.json isn't needed after a restart: everything above the LIB is printed again, and firecore ignores what it already has.
• Stop pre-marking the last fired block as fired on restart, so it is printed again.
• Optionally print FIRE CURSOR for a poller with position state beyond the block number.

Each chain's poller then:

1. picks up the new blockpoller;
2. accepts the new flag;
3. updates its documented --reader-node-arguments to --reader-node-cursor plus --resume-lib={resume-lib-num}, keeping {first-streamable-block} for a fresh start.

Readers that sit on a node, such as instrumented geth, would not use this feature: the node keeps its own position in its database.

Documentation to update

Not in this PR. In streamingfast/firehose-docs:

• references/firehose-protocol.md: add FIRE CURSOR to the message types. FIRE SIGNAL, which dummy-blockchain prints, isn't documented there either.
• references/cli/reader-node.md: the --reader-node-cursor flag, the six templates and that they are read on every start, and reader-cursor.json in the working directory.
• integrate-new-chains/integration-overview.md (RPC Poller) and firehose-starter.md: resume from the reader cursor, not the reader's own state.
• firehose-setup/ poller pages (injective, stellar, starknet, tron, avalanche): their --reader-node-arguments examples, once each poller supports it.
• Optionally architecture/components/reader.md, architecture/data-flow.md and architecture/components/merger.md.

Elsewhere:

• blockpoller has no README. Its resume behaviour is documented only in code, so the poller follow-up is a good moment for a package doc.
• firehose-acme's INTEGRATION.md could show the flag once dummy-blockchain uses it. dummy-blockchain is a node, not a poller, and needs no change.

Testing

Test code in this PR

• Unit tests, run by go test ./...:
  • node-manager/mindreader/cursor_test.go: the cursor tracker;
  • consolereader_test.go: FIRE CURSOR parsing;
  • cmd/apps/reader_node_test.go: the templates and detecting them.
• devel/reader-cursor/, a runnable example. Its scripts start firecore and take minutes, so they are run by hand rather than by go test.
  • fakechain/: the synthetic chain.
  • fakereader/: a reference reader, using both the old and the new way of resuming.
  • checkmerged/: a merged-blocks checker.
  • kill-test.sh, restore-test.sh: the scripts.
  • README.md: how to run them, with the expected results.

Results

Unit tests:

• The tracker: out-of-order uploads; values arriving before, during and after their block is stored; a block printed twice; a restart; Close; values for blocks never queued.
• The templates.
• FIRE CURSOR parsing.
• go test ./... passes; node-manager/mindreader also passes with -race.

Kill tests, devel/reader-cursor/. A synthetic chain is read by a reference reader under firecore start reader-node merger, with local stores. The scripts build firecore and the example from the checkout, and the README explains how to run them.

• Each run SIGKILLs firecore and the reader together 8 times, at random moments 2–5 s in, then runs to the target and checks the merged blocks.
• The reader prints as fast as firecore reads, with 2 KB blocks.

Reader	Restarts from	Result
can produce any block again, marks blocks done once printed	after its last printed block	merger stuck at 19,699
same	{resume-lib-num} + 1	blocks 1–42,899, no gap
state only moves forward, commits every 500 blocks independently	its last commit, printing only after its last printed block	merger stuck at 17,599; the missing blocks need an older state
same, committing only what {cursor-file} shows stored	its last commit, printing after {resume-block-num}	blocks 1–30,399, no gap, every block's state follows from the previous one. At each kill the committed state was at the cursor, never past it. A commit waited about 0.6 s, mostly the 500 ms upload pass
same, state put past the cursor on purpose	—	refuses to start ("restore a snapshot at or before 7000"). After a snapshot at 6,000 it indexes again to 7,000, checks its state against {resume-cursor}, and continues to 12,399 with no gap

Not covered

• Power loss: a file:// one-block store doesn't fsync its files, so it can lose stored blocks older than the cursor. Object stores are durable once the upload returns.
• Fail fast on gaps: a check in the console reader or the mindreader that each block links to the previous one, which would turn a gap into an immediate error instead of a stuck merger. Not in this PR.
• A reader restarted while firecore keeps running: for example through the node manager API. The templates are re-read for it, but this wasn't tested.

Reviewing

1. Read the code. The core is node-manager/mindreader/cursor.go, about 300 lines. The rest is wiring:
   • the archiver queues each block before writing its local file, and the uploader reports each completed upload;
   • ConsoleReader.readCursor parses FIRE CURSOR;
   • the templates and --reader-node-cursor are in cmd/apps/reader_node.go;
   • Superviser.SetArgumentsFunc re-resolves the arguments on each start.
2. Without the flag, nothing changes: the tracker isn't created, the templates aren't resolved, and the console reader just parses FIRE CURSOR lines that no handler receives.
3. Run the tests:
   • go test ./...
   • go test -race ./node-manager/mindreader/
4. Run the example: devel/reader-cursor/kill-test.sh rewind legacy and then rewind cursor, about 2 minutes each. The first must end with the merger stuck, the second with no gap. The stateful runs and restore-test.sh cover a reader whose state only moves forward.
5. Decide on naming: the flag name, the template names, the FIRE CURSOR format and the file's JSON fields become an interface for every reader, so they're worth agreeing on before merge.

Next steps

If this direction looks right, follow-up PRs can be provided:

• the blockpoller change above;
• the per-chain pollers that use it;
• the firehose-docs updates listed above.

Changes to the names or the file format before merge are welcome too.

[maoueh]

Not completely against the feature but I wouldn't say it convinces me neither.

We have seen in the past that writing a cursor each block without proper care could lead to long backprocessing due to sync writes blocking stdout I/O for example.

I'm far from being convinced on new added FIRE CURSOR in the protocol. It seems to me that FIRE BLOCK is already doing the same job and not sure I see why the reader node would drive two signals.

Maybe we introduce instead a "sync" mode with a write ahead log instead for example, that would avoid all this cursoring needs.

Maybe we add a mode automatically resume to "earliest" missing one blocks on bootstrap, that would be the second part that auto-recover poller which is less "controversial" if I can say it like that.

[maoueh]

Converting to draft until we settle on the correct path forward.

Is what you are trying to achieve is auto-recovering poller nodes?

[matthewdarwin]

@maoueh Thanks for reviewing.

Claude as usual was focused on fixing the specific problem of making the cursor file survive a specific failure path, rather than looking holistically at "auto-recover poller nodes", but yes, "auto-recover poller nodes" is a better solution. eg figure out what block the merger has last merged and then start there... there are a couple of things that would be improved with such a feature.

Although in the specific case I was working on, the "poller" can't rewind, so it would need to stop (or have some workflow) that rolls back to a previous snapshots. (necessary workflow is operator responsibility, not a firehose one... firehose just need to provide the correct signals that can trigger the right behaviour... eg merger is at X, you're at Y, there is a gap, do something about it).

The big feature that would be helpful is if the --start-block for the reader could say and know where to start from. Especially good for spinning up a new poller node in k8s Also great for the remote reader too as a fallback to another provider (our firehose is lagging, start reading from other provider starting from last merged block)

Happy to work on alternate solutions if helps or if y'all take the leadership such features. Let me know how I can best help.

[maoueh]

Here different strategies I see:

• Direct poller implementation, instead of going through the stdout dance, poller inherits from firehose-core "reader" as a library and writes directly to one-block files and advertise, a cursor can be managed automatically if desired on the one-block write notif, poller restarts on cursor.
• Or still using the stdout protocol, but now the poller still gain a cursor on its implementation, the poller "query" firehose stack for latest one-block written, could need a firehose-core to record such signal.

In both case poller is modified to restart from last written block.

This use side-channel to move the "write" cursor on the poller side avoiding completely the need to modify Firehose protocol.