Repository navigation
feat(autobahn): prepare a restored node's epoch registry from a checkpoint - #4860
blindchaser wants to merge 15 commits into
Conversation
KVImporter now holds at most 256 MiB of key and value bytes that are not yet flushed. When the budget is full, AddNode waits for a flush to return bytes. Each worker also flushes when it holds budget/16 bytes, so the four workers together hold less than a quarter of the budget, and a node larger than half the budget is refused. A waiting AddNode therefore always waits on bytes that are in a channel or in a flush, and those bytes always come back, so the import cannot deadlock. The worker threshold is 16 MiB, a quarter of the default 64 MiB memtable. Pebble keeps a batch of half a memtable or more on the Go heap until it is written to disk, after the commit that returns its bytes to the budget. In an import of 512 MiB of 24 KiB rows, the peak live heap was 0.7 to 1.0 GiB with a 32 MiB threshold and 0.4 GiB with 16 MiB. An error broadcasts under the budget lock, so a waiting AddNode cannot miss it. A release that takes the count below zero fails the import instead of being hidden. Co-authored-by: Cursor <cursoragent@cursor.com>
ExportSnapshotDir streams the live rows of a closed snapshot dir in physical-key order, merging the four data databases. SnapshotDirStats returns what the snapshot metadata records: the row count, the key and value bytes, and the state hash, which is the checksum of the store-wide LtHash. The export counts the rows and bytes it emits, and fails when they differ from the metadata. ReadSnapshotMiscValue reads one non-EVM row. It fails when the dir has no misc database, so a wrong path is not read as a snapshot without the row. The state-sync snapshot tool uses these functions on a staged checkpoint where no CommitStore is open. Co-authored-by: Cursor <cursoragent@cursor.com>
KVImporter now holds at most 256 MiB of key and value bytes that are not yet flushed. When the budget is full, AddNode waits for a flush to return bytes. Each worker also flushes when it holds budget/16 bytes, so the four workers together hold less than a quarter of the budget, and a node larger than half the budget is refused. A waiting AddNode therefore always waits on bytes that are in a channel or in a flush, and those bytes always come back, so the import cannot deadlock. The worker threshold is 16 MiB, a quarter of the default 64 MiB memtable. Pebble keeps a batch of half a memtable or more on the Go heap until it is written to disk, after the commit that returns its bytes to the budget. In an import of 512 MiB of 24 KiB rows, the peak live heap was 0.7 to 1.0 GiB with a 32 MiB threshold and 0.4 GiB with 16 MiB. An error broadcasts under the budget lock, so a waiting AddNode cannot miss it. A release that takes the count below zero fails the import instead of being hidden. Co-authored-by: Cursor <cursoragent@cursor.com>
ExportSnapshotDir streams the live rows of a closed snapshot dir in physical-key order, merging the four data databases. SnapshotDirStats returns what the snapshot metadata records: the row count, the key and value bytes, and the state hash, which is the checksum of the store-wide LtHash. The export counts the rows and bytes it emits, and fails when they differ from the metadata. ReadSnapshotMiscValue reads one non-EVM row. It fails when the dir has no misc database, so a wrong path is not read as a snapshot without the row. The state-sync snapshot tool uses these functions on a staged checkpoint where no CommitStore is open. Co-authored-by: Cursor <cursoragent@cursor.com>
…to cover them The layout now lists the block DB, receipt store, state store and app-hash vault dirs beside FlatKV, so state sync can require them empty on restore. It takes the storage config and the genesis doc, because the storage config requires the initial height. The new test opens the storage manager the way node startup does, and fails on any entry under the persistent state dir that the layout does not cover. Co-authored-by: Cursor <cursoragent@cursor.com>
PrepareEpochRegistry reads the epoch_registry row from a staged FlatKV checkpoint, the PersistedEpochRegistry that consensus commits in EpochState, and writes it as an epoch registry snapshot in a staging dir. The registry must open from genesis the same way node startup opens it, through p2p.OpenEpochRegistry, which BuildDataState now also uses. It must hold the epoch of the checkpoint height and the epoch after it, and a height at the last road of an epoch is refused. With no row the registry holds the genesis epochs, and an empty row is an error. The genesis check binds a row to this chain only while the row holds epoch 0 or 1. InstallEpochRegistry moves the staged snapshot to the layout's EpochRegistryDir. The layout also names the state sync proof file that a restore installs. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4860 +/- ##
==========================================
+ Coverage 61.59% 61.64% +0.04%
==========================================
Files 2079 2082 +3
Lines 161891 162138 +247
==========================================
+ Hits 99715 99946 +231
- Misses 62171 62187 +16
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…seed Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # sei-tendermint/internal/evmonlyapp/app_test.go
readCommittedVersion and the snapshot dir readers held the same read-only pebble.Open. Both now call openPebbleReadOnly, which returns the Pebble error unchanged, so the error text and the ErrDBDoesNotExist check in readCommittedVersion are the same as before. Co-authored-by: Cursor <cursoragent@cursor.com>
A flush returns its rows' bytes to the import byte budget, but resetting ltMutations to length zero kept the rows reachable through the backing array. A byte-triggered flush can be shorter than the batch before it, so the next batch did not overwrite every slot, and rows that the budget no longer counted stayed in memory. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # sei-db/state_db/sc/flatkv/import_export_test.go # sei-db/state_db/sc/flatkv/snapshot_read.go
State the crash contract of InstallEpochRegistry, return the genesis validators in key order, and rename the test for a row pruned past genesis so that its name states the limit. Co-authored-by: Cursor <cursoragent@cursor.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
PR SummaryHigh Risk Overview
On the FlatKV side, new Extensive tests cover layout coverage, registry edge cases, restored-node anchoring, and importer/snapshot parity. Reviewed by Cursor Bugbot for commit 06bbf5a. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 988890c0d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Builds a restored node's epoch registry from the FlatKV checkpoint's epoch_registry row, and makes AutobahnLayout list every store dir that startup opens by building the storage config through the same function startup uses. The crash-retry path of InstallEpochRegistry, which its own doc tells callers to use, can wipe the installed registry and still report success, so this is request_changes. That finding came from codex and I confirmed it against the tree; it was the only reading, and the startup refactor looks behavior-preserving. There is no Go toolchain in this sandbox, so the tests were not run.
Blocking
1 finding on the changed lines, as inline comments.
seidroid review · decision request_changes · session 1ae92a41958849c4af27d417a5ce2086 · turn resp_claude_b4087d7fe11f925b667305594404e0a7 · item 93ee4fbb60d75f138e58637962544178
Findings: 1 blocking | 0 non-blocking | 1 posted inline
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 988890c. Configure here.
InstallEpochRegistry removed the installed snapshot before it checked the staged one, and ignored a missing source, so a second call with the same staging dir left only the genesis epochs and reported success. It now refuses a missing staged snapshot before it removes anything. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
InstallEpochRegistry now checks that the staging dir holds a prepared snapshot before it removes the installed one, and it no longer swallows ErrNotExist. A new test shows a second install from the same staging fails and keeps the installed registry, so my blocker is fixed and nothing else blocks. Nothing else in the diff changed since my last review; codex found nothing in this round, which matches my reading, and there is still no Go toolchain here, so the tests were not run.
seidroid review · decision approve · session 1ae92a41958849c4af27d417a5ce2086 · turn resp_claude_8354077b2317694be5ea1786aa0eda4d · item 62ee35e828c25299a930ac6fb0647504
Findings: 0 blocking | 0 non-blocking | 0 posted inline
Superseded: the latest review found nothing blocking in this change.
…omments Co-authored-by: Cursor <cursoragent@cursor.com>
| ) | ||
|
|
||
| // SnapshotDir returns the dir under stateDir that holds the epoch registry snapshot. | ||
| func SnapshotDir(stateDir string) string { |
There was a problem hiding this comment.
Why are we reading a file from directory? I suppose writing and reading of epoch snapshots should all go through storage manager?
The previous WAL files are temporary measure, they should go away very soon, restored by the values in FlatKV.
…alue Add epoch.NewRegistryFromSnapshot, which opens a registry from the FlatKV epoch_registry value with the same checks as NewRegistry and persists it when a state dir is set. Remove epoch.RestoreSnapshot. node.OpenCheckpointEpochs opens the checkpoint registry in memory, and InstallEpochRegistry writes it into the persistent state dir with the new constructor. There is no staging dir and no rename, and an install can run again with the same CheckpointEpochs. Co-authored-by: Cursor <cursoragent@cursor.com>

Summary
A node restored from a FlatKV snapshot at height H must know the committees of the epochs around H before it can verify the next blocks. This PR builds that epoch registry from the
epoch_registryrow in the snapshot. It also makesAutobahnLayoutlist every store dir that node startup opens, soseid statesync importcan require all of them to be empty.How it works:
epoch.NewRegistryFromSnapshotopens a registry from the row value. It runs the same checks thatNewRegistryruns on a loaded snapshot, and with a state dir it persists the row there. The decode and the file format stay in theepochpackage.node.OpenCheckpointEpochsreads the row from the FlatKV snapshot dir and opens the registry in memory. It must hold the epoch of H and the next epoch, and an H on the last road of an epoch is refused.node.InstallEpochRegistryremoves<psd>/epochsand writes the row there with the same constructor. It can run again with the same row, so a rerun after a crash is safe.autobahnStorageConfig, so the layout lists the dirs that startup really opens.For reviewers
NewRegistryFromSnapshotinepoch/registry.go. Is the name good? Should the layout keepepoch.SnapshotDirfor the<psd>/epochspath?node/setup.goandp2p/giga_router_common.godoes not change behavior.Test plan
NewRegistryreads back over an older snapshot, the genesis of another chain, a row pruned past genesis, and a malformed row that writes nothing.scripts/ramtest.sh -raceonnode,autobahn/epoch,autobahn/data,evmonlyappandp2p;make fmtcheckand golangci-lint v2.14.0 are clean.