[Storehouse] Fast Storehouse Bootstrap via Compacted Sealed State

[zhangchiqing]

Problem

Storehouse (register store on disk) is enabled in two phases: first --enable-background-storehouse-indexing=true runs a background indexer that populates the storehouse database with all registers up to the latest height; once caught up, the node is restarted with --enable-storehouse=true (see cmd/execution_config.go:156-157).

Today the background indexer always starts from the root block using the root checkpoint. For an execution node that has been running for a long time without storehouse, catching up from the root block to the latest height takes a very long time.

This is unnecessary work, because block execution does not need registers of past blocks to be indexed.

It is sufficient to bootstrap the storehouse database with the register state of the last sealed and executed block using ImportRegistersFromCheckpoint, and have the background indexer start indexing from that height instead of the root block.

Why this is currently hard

ImportRegistersFromCheckpoint reads leaf nodes via OpenAndReadLeafNodesFromCheckpointV6, which requires the checkpoint file to contain exactly one trie whose root hash matches the expected state commitment (see ledger/complete/wal/checkpoint_v6_leaf_reader.go:45).

However, checkpoints produced during normal operation by the checkpointer contain up to 500 tries (see ledger/complete/wal/checkpointer.go:316), so they cannot be consumed by ImportRegistersFromCheckpoint.

Producing a usable single-trie checkpoint today is a manual, error-prone, multi-step procedure that requires downtime (execution and ledger services stopped):

1. Read the last sealed and executed block and its state commitment with read-protocol-state blocks --executed (cmd/util/cmd/read-protocol-state/cmd/blocks.go:47).
2. Extract a v6 checkpoint containing the single trie for that state commitment with execution-state-extract --no-migration (cmd/util/cmd/execution-state-extract/cmd.go). The output file is always named root.checkpoint.
3. The checkpoint must be named after the latest WAL segment so that ledger startup replays correctly. But the latest WAL segments usually contain trie updates for blocks that are executed but not yet sealed, so the WAL must be trimmed so that the trie update for the last sealed and executed block is the last record. This is done with find-trie-root --trim-as-latest-wal=true (cmd/util/cmd/find-trie-root/cmd.go:39). Note: it currently scans segments from the first WAL file forward, which is slow; the target trie root is near the end, so scanning backwards from the last segment would be much faster.
4. Rename root.checkpoint (and its 17 sub-files) to checkpoint.<segment> matching the trimmed WAL segment number, using tools/move-checkpoint.sh.
5. Roll back the executed height to the last sealed and executed block. After trimming, tries for executed-but-unsealed blocks are gone from the ledger, but the execution database still records those blocks as executed; without a rollback the node would resume from lastExecuted + 1 and fail to find the parent state commitment in the ledger.

Proposed solution

Definition: Compacted Sealed State

An execution state directory is in the Compacted Sealed State when, for S = the last sealed and executed block:

1. The latest checkpoint file checkpoint.N contains exactly one trie, whose root hash equals the state commitment of S.
2. WAL segment N is the latest non-empty segment, and its last trie-update record is the one producing the state commitment of S. Any segments newer than N are removed (backed up).
3. The executed height recorded in the execution database equals the height of S.

In this state, ledger startup, storehouse register import, and block execution all agree on a single anchor point: the last sealed and executed block.

New util: compact-execution-state

A new offline subcommand under cmd/util/cmd/ that turns a stopped EN into the Compacted Sealed State in a single run.

Ordering invariant: all steps that mutate the execution state directory happen after the checkpoint extraction has fully succeeded. State extraction rebuilds the trie in memory (memory footprint is acceptable on production ENs) and writes the checkpoint to a scratch/output directory outside the state dir. If the run is interrupted during or before extraction, the state dir is untouched and the node can simply be restarted.

Internally it orchestrates:

1. Resolve anchor (read-only): open the protocol/execution database read-only, find the last sealed and executed block S and its state commitment C (reusing logic from read-protocol-state blocks --executed).
2. Locate WAL segment (read-only): scan WAL segments backwards from the last segment to find the segment N containing the trie update with root hash C (extending find-trie-root logic).
3. Extract single-trie checkpoint (read-only w.r.t. state dir): build the trie for C from the existing checkpoint + WAL replay in memory and write a v6 checkpoint containing only that trie to a scratch directory (reusing execution-state-extract --no-migration logic, skipping migrations and reports).
4. Trim WAL (first mutation): write a trimmed replacement of segment N whose last record is the trie update for C; move the original segment N and all newer segments to a backup directory (reusing find-trie-root --trim-as-latest-wal logic).
5. Name checkpoint after segment: move the checkpoint and its 17 sub-files from the scratch directory into the execution state dir as checkpoint.N (reimplementing move-checkpoint.sh in Go).
6. Roll back executed height: set the executed height to S (reusing rollback-executed-height logic) so the node re-executes unsealed blocks after restart.
7. Verify: re-open the produced checkpoint.N and assert it has a single root hash equal to C (the same check ImportRegistersFromCheckpoint performs); assert no non-empty WAL segment newer than N exists; assert executed height equals height of S. Fail loudly if any check fails — the backup directory allows manual recovery.

Backed-up WAL segments are kept after successful verification; cleanup is left to the operator.

The command must refuse to run against a live node (e.g., detect DB locks) and must require an explicit, empty backup directory, following the same safety conventions as find-trie-root.

Storehouse bootstrap mode

When the storehouse database is empty, the EN must choose where to bootstrap it from. Both behaviors remain available, selected by a single enum flag in the EN config (mutually exclusive modes of one decision — an enum avoids the invalid "both set" combination that two boolean flags would allow):

--storehouse-bootstrap-mode=root-checkpoint    # default, current behavior:
                                               # start from the root block using the
                                               # root checkpoint
--storehouse-bootstrap-mode=sealed-checkpoint  # import registers from the latest
                                               # single-trie checkpoint (Compacted Sealed
                                               # State) via ImportRegistersFromCheckpoint,
                                               # then index forward from that height

The flag applies to both ways storehouse indexing can run, which is why it is not scoped to the background indexer:

• With background indexing (--enable-background-storehouse-indexing=true): the background indexer bootstraps the empty storehouse DB according to the mode, then indexes forward while the node keeps executing blocks as usual (no downtime beyond the compaction run).
• Without background indexing (--enable-storehouse=true on an empty storehouse DB): the node bootstraps the storehouse DB according to the mode during startup. In sealed-checkpoint mode, the EN does not execute any block until the register import and catch-up indexing are done; block execution starts only once the storehouse is ready. This trades availability for operational simplicity — one restart, no separate background-indexing phase.

Common rules:

• The flag only takes effect when the storehouse database is empty; on a non-empty database indexing resumes from its last indexed height as today.
• No silent fallback: if sealed-checkpoint is selected but the latest checkpoint is not a valid single-trie checkpoint whose root hash matches the last sealed and executed block's state commitment, the node refuses to start with a clear error instead of falling back to root-checkpoint (which would silently begin a multi-day catch-up).
• The Access Node needs no flag: this flag lives in the EN config (cmd/execution_config.go), and the AN has no concept of executed blocks, so its register indexing keeps the root-checkpoint bootstrap unconditionally. The shared import logic (ImportRegistersFromCheckpoint) stays parameterized by height/checkpoint-file/root-hash so both node roles use it.

Operational flow (target)

Path A — background indexing (minimal downtime):

1. Stop the execution node (and ledger service if running separately). Downtime begins.
2. Run compact-execution-state — the node's state dir is now in the Compacted Sealed State.
3. Restart the node with --enable-background-storehouse-indexing=true --storehouse-bootstrap-mode=sealed-checkpoint. Downtime ends.
4. The indexer imports registers from checkpoint.N at the sealed height and indexes forward while the node keeps executing blocks; catch-up now only covers blocks executed since the compaction, not the whole chain history.
5. Once caught up, restart with --enable-storehouse=true.

Path B — direct enablement (simpler, longer downtime):

1. Stop the execution node. Downtime begins.
2. Run compact-execution-state.
3. Restart the node with --enable-storehouse=true --storehouse-bootstrap-mode=sealed-checkpoint.
4. The node imports registers from checkpoint.N, catches up indexing, and only then starts executing blocks. Downtime (for block execution) ends when the bootstrap completes — which is short, since only blocks executed since the compaction need indexing.

TODO

• T1 — Backwards WAL scan. Extend find-trie-root (or the shared logic it uses) to scan segments from the last segment backwards, since the target trie root is near the end. Per-segment records are still read forward; only the segment iteration order is reversed.
• T2 — New compact-execution-state util. New subcommand under cmd/util/cmd/ orchestrating steps 1–7 above, honoring the ordering invariant (no state-dir mutation before extraction succeeds). Refactor reusable pieces out of read-protocol-state, find-trie-root, execution-state-extract, and rollback-executed-height rather than duplicating them.
• T3 — Checkpoint naming in Go. Reimplement the move-checkpoint.sh rename (header + 17 sub-files) as a Go function shared by the util.
• T4 — Verification step. Implement the post-run checks (single root hash matches C, no newer non-empty WAL, executed height == sealed height).
• T5 — Storehouse bootstrap mode flag. Add --storehouse-bootstrap-mode (root-checkpoint | sealed-checkpoint, default root-checkpoint) to the EN config. In sealed-checkpoint mode with an empty storehouse DB, bootstrap via ImportRegistersFromCheckpoint from the latest single-trie checkpoint and index forward from the last sealed and executed height; fail node startup (no silent fallback) if the checkpoint is missing or invalid. Applies both to background indexing and to direct storehouse enablement.
• T5a — Hold execution during direct bootstrap. When --enable-storehouse=true is set on an empty storehouse DB (no prior background indexing), gate block execution on storehouse readiness: the node performs the register import and catch-up indexing first, and only then starts executing blocks.
• T6 — Tests. Unit tests for WAL trimming (including multi-segment and already-at-boundary cases), checkpoint extraction + naming, verification failures, and an end-to-end test: run an EN state fixture through compaction, then bootstrap storehouse and index forward.
• T7 — Runbook. Operator documentation for the downtime procedure, including recovery from the backup directory if a step fails.

Resolved design decisions

• Backed-up WAL segments are kept after successful verification; the operator cleans them up.
• Bootstrap source is user-selectable via the single enum flag --storehouse-bootstrap-mode (see above). The EN supports both modes — with background indexing (node keeps executing while indexing catches up) or without it (node holds block execution until the bootstrap completes). The AN always uses the root-checkpoint bootstrap since it has no concept of executed blocks.
• In-memory trie rebuild is acceptable: the memory footprint of execution-state-extract style extraction is fine on production ENs. The critical constraint is ordering — WAL trimming (and all other state-dir mutations) must happen only after extraction has fully succeeded, so an interrupted extraction leaves the state dir unchanged.