Skip to content

Commit 53168c2

Browse files
seidb: add changelog mode and --inspect-plan to speed up EVM digest (#4419)
The kvrepair drill-down runs `seidb evm-logical-digest` many times at one height on a reserve and on a production node: one digest, then one `--inspect-bucket --list` report for each bucket and key prefix. The comparison height is almost never a memIAVL snapshot height, so each run uses `--memiavl-open-mode replay`. That mode opens the full DB, catches up every module through the changelog, and walks the EVM tree with random reads, and it does this again for each report. On a mainnet node the gap between snapshots is many thousands of blocks, so the drill-down is slow. This PR adds two flags and does not change any report value. `--memiavl-open-mode changelog` scans the `kvs` file of the newest snapshot at or below `--height` and merges in a sorted overlay of the EVM writes from the changelog versions above it (`memiavl.ReplayTreeChangelog`, `scanMemiavlChangelogEVMLeaves`). It reports the same rows and the same mode labels as replay (`semantic-replay`, `translator-replay`), so `kvrepair-export` accepts its reports. It refuses a changelog that does not reach the height, that starts above the snapshot, or that upgrades the `evm` tree in that range. The stream opens the snapshot `kvs` file before the scan, so a prune by the running node does not remove the data under it. When the node prunes the selected snapshot before the stream opens, the error wraps `memiavl.ErrSnapshotPruned`, and changelog mode selects the snapshot again one time. `--inspect-plan` reads a JSON list of inspect reports and takes all of them from one scan (`inspectFanout`); each report is byte-equal to the report of a single run with the same flags. It refuses two reports that write one file, by device and inode, so a relative path, a symlink, or a hard link does not get past the check. To make this possible, the memiavl and composite paths now go through one leaf stream (`memiavlLeafStream`), account inspect lists come out in key order (before, `--list-limit` kept a subset in map order), and `entryHash` no longer allocates for account and storage rows. The EVM logical tools (`evm-logical-digest`, `--inspect-plan`, and `kvrepair-export`) move into a new package, `sei-db/tools/cmd/seidb/operations/evmlogical`. The command names and flags do not change. The FlatKV bucket names and the physical-key classifier stay in `operations` and are exported, so both packages use the same copy. The `evmlogical` tests have their own copy of some small test-data helpers (`addrN`, `noncePair`, `newTestMemiavlStore`, and others), because there is no shared test-helper package for `operations`. The first commit adds golden tests that pin the digest and inspect JSON for every backend, normalization, and open mode on the current code. No later commit changes an old golden case. The only output text that changes is descriptive: the `normalization` field of semantic and translator reports in replay mode now has the same text as in snapshot mode ("raw memiavl EVM keys" in place of "replayed memiavl EVM keys"), and changelog reports have their own `source` text. Review `changelog_range.go` most closely, mainly the version and upgrade checks. The tool is read-only and has no consensus or state impact. Validation: ramtest (also with `-race`) for `sei-db/tools/...` and `sei-db/state_db/sc/memiavl/...`, with tests that compare changelog mode to replay mode at many heights, including the kvrepair export end-to-end test in both modes. On a harbor chain (4 composite validators and 1 memIAVL reserve, snapshot at 20000), every changelog report was equal to the replay report at heights 15000 to 28000: 74 pairs of digest, bucket list, and shard reports over two runs. The reserve and composite digests at 28000 were equal, and `kvrepair-export` accepted the changelog reports. The EVM overlay was about 1 key for each block on that chain (8,260 keys, 83 KB for 8,000 blocks). On two pacific-1 shadow nodes at height 235215000, a memIAVL-only node 85,000 blocks above its snapshot and a node with a complete migration, the changelog and replay reports were equal on each node, the two nodes had the same final digest over 707,583,826 rows, and the `--inspect-plan` reports were byte-equal to single runs. On the memIAVL node, the changelog digest took 405 s with a 24 GB peak RSS, and the replay digest took 2,286 s with a 142 GB peak RSS. A narrow storage list took 120 s, and 10,052 s with replay. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 84852aa commit 53168c2

19 files changed

Lines changed: 7472 additions & 448 deletions
Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
package memiavl
2+
3+
import (
4+
"errors"
5+
"fmt"
6+
"io/fs"
7+
"math"
8+
"os"
9+
"path/filepath"
10+
11+
"github.com/sei-protocol/sei-chain/sei-db/common/utils"
12+
"github.com/sei-protocol/sei-chain/sei-db/proto"
13+
"github.com/sei-protocol/sei-chain/sei-db/wal"
14+
)
15+
16+
// TreeChangelogRange is the snapshot a tree changelog replay starts from, and the version it reaches.
17+
type TreeChangelogRange struct {
18+
// SnapshotDir is the snapshot directory the replay starts from.
19+
SnapshotDir string
20+
// SnapshotVersion is the version of the snapshot at SnapshotDir.
21+
SnapshotVersion int64
22+
// StartVersion is the first version the replay applies above the snapshot.
23+
StartVersion int64
24+
// Version is the version the replay reaches. It is SnapshotVersion when nothing is replayed.
25+
Version int64
26+
}
27+
28+
// ErrSnapshotPruned reports that a snapshot a reader selected was pruned before the reader
29+
// finished with it or with the changelog above it.
30+
var ErrSnapshotPruned = errors.New("memiavl snapshot was pruned while it was read")
31+
32+
// SnapshotPrunedError returns err wrapped with ErrSnapshotPruned when snapshotDir no longer
33+
// exists, and err unchanged otherwise.
34+
func SnapshotPrunedError(snapshotDir string, err error) error {
35+
if err == nil {
36+
return nil
37+
}
38+
// The node truncates the changelog only below its earliest snapshot, so the entries above
39+
// snapshotDir can go missing only after snapshotDir itself is removed.
40+
if _, statErr := os.Stat(snapshotDir); !errors.Is(statErr, fs.ErrNotExist) {
41+
return err
42+
}
43+
return fmt.Errorf("%w: %s: %w", ErrSnapshotPruned, snapshotDir, err)
44+
}
45+
46+
// ReplayTreeChangelog passes to fn, in changelog order, every changeset of treeName that
47+
// OpenDB(targetVersion) would apply on top of its snapshot. A targetVersion of 0 selects the
48+
// current snapshot and the changelog tip. It loads no tree and never repairs the changelog.
49+
//
50+
// It refuses a changelog that starts above the first version after the snapshot, a changelog
51+
// that ends below targetVersion, and an upgrade that adds, deletes, or renames treeName. A
52+
// failure caused by the node pruning the selected snapshot wraps ErrSnapshotPruned.
53+
func ReplayTreeChangelog(
54+
dir string,
55+
targetVersion int64,
56+
treeName string,
57+
fn func(version int64, changeSet proto.ChangeSet) error,
58+
) (TreeChangelogRange, error) {
59+
snapshotDir, err := treeChangelogSnapshotDir(dir, targetVersion)
60+
if err != nil {
61+
return TreeChangelogRange{}, err
62+
}
63+
r, err := replayTreeChangelogFrom(dir, snapshotDir, targetVersion, treeName, fn)
64+
if err != nil {
65+
return TreeChangelogRange{}, SnapshotPrunedError(snapshotDir, err)
66+
}
67+
return r, nil
68+
}
69+
70+
// replayTreeChangelogFrom is ReplayTreeChangelog from the snapshot at snapshotDir.
71+
func replayTreeChangelogFrom(
72+
dir, snapshotDir string,
73+
targetVersion int64,
74+
treeName string,
75+
fn func(version int64, changeSet proto.ChangeSet) error,
76+
) (TreeChangelogRange, error) {
77+
metadata, err := readMetadata(snapshotDir)
78+
if err != nil {
79+
return TreeChangelogRange{}, fmt.Errorf("read snapshot metadata %s: %w", snapshotDir, err)
80+
}
81+
if metadata.InitialVersion < 0 || metadata.InitialVersion > math.MaxUint32 {
82+
return TreeChangelogRange{}, fmt.Errorf("invalid initial version: %d", metadata.InitialVersion)
83+
}
84+
r := TreeChangelogRange{
85+
SnapshotDir: snapshotDir,
86+
SnapshotVersion: metadata.CommitInfo.Version,
87+
StartVersion: utils.NextVersion(metadata.CommitInfo.Version, uint32(metadata.InitialVersion)),
88+
Version: metadata.CommitInfo.Version,
89+
}
90+
91+
stream, err := wal.NewChangelogWAL(utils.GetChangelogPath(dir), wal.Config{NoRepairOnOpen: true})
92+
if err != nil {
93+
return TreeChangelogRange{}, fmt.Errorf("open changelog: %w", err)
94+
}
95+
defer func() { _ = stream.Close() }()
96+
span, err := readChangelogSpan(stream)
97+
if err != nil {
98+
return TreeChangelogRange{}, err
99+
}
100+
101+
endVersion, err := span.replayEnd(r.StartVersion, targetVersion)
102+
if err != nil {
103+
return TreeChangelogRange{}, fmt.Errorf("replay above snapshot %d: %w", r.SnapshotVersion, err)
104+
}
105+
if endVersion < r.StartVersion {
106+
return r, nil
107+
}
108+
if err := replayTreeChangeSets(stream, span, r.StartVersion, endVersion, treeName, fn); err != nil {
109+
return TreeChangelogRange{}, err
110+
}
111+
r.Version = endVersion
112+
return r, nil
113+
}
114+
115+
// treeChangelogSnapshotDir returns the snapshot directory OpenDB(targetVersion) loads.
116+
func treeChangelogSnapshotDir(dir string, targetVersion int64) (string, error) {
117+
if targetVersion == 0 {
118+
name, err := os.Readlink(currentPath(dir))
119+
if err != nil {
120+
return "", fmt.Errorf("read current snapshot link: %w", err)
121+
}
122+
return filepath.Join(dir, name), nil
123+
}
124+
snapshotVersion, err := seekSnapshot(dir, targetVersion)
125+
if err != nil {
126+
return "", fmt.Errorf("fail to seek snapshot: %w", err)
127+
}
128+
return filepath.Join(dir, snapshotName(snapshotVersion)), nil
129+
}
130+
131+
// changelogSpan is the version range a changelog holds.
132+
type changelogSpan struct {
133+
// empty is true for a changelog with no entries; the other fields are then zero.
134+
empty bool
135+
firstVersion int64
136+
lastVersion int64
137+
// delta is the version of an entry minus its changelog index.
138+
delta int64
139+
}
140+
141+
// readChangelogSpan returns the version range of stream.
142+
func readChangelogSpan(stream wal.ChangelogWAL) (changelogSpan, error) {
143+
delta, hasEntries, err := computeWALIndexDelta(stream)
144+
if err != nil {
145+
return changelogSpan{}, fmt.Errorf("compute changelog index delta: %w", err)
146+
}
147+
if !hasEntries {
148+
return changelogSpan{empty: true}, nil
149+
}
150+
firstIndex, err := stream.FirstOffset()
151+
if err != nil {
152+
return changelogSpan{}, fmt.Errorf("read changelog first index: %w", err)
153+
}
154+
lastIndex, err := stream.LastOffset()
155+
if err != nil {
156+
return changelogSpan{}, fmt.Errorf("read changelog last index: %w", err)
157+
}
158+
if lastIndex < firstIndex {
159+
return changelogSpan{empty: true}, nil
160+
}
161+
// #nosec G115 -- changelog indexes are far below MaxInt64
162+
return changelogSpan{firstVersion: int64(firstIndex) + delta, lastVersion: int64(lastIndex) + delta, delta: delta}, nil
163+
}
164+
165+
// replayEnd returns the last version to replay from startVersion for targetVersion, where 0 means
166+
// the changelog tip. A result below startVersion means there is nothing to replay. It refuses a
167+
// span that does not hold every version from startVersion to the result.
168+
func (s changelogSpan) replayEnd(startVersion, targetVersion int64) (int64, error) {
169+
if targetVersion == 0 {
170+
if s.empty || s.lastVersion < startVersion {
171+
return startVersion - 1, nil
172+
}
173+
targetVersion = s.lastVersion
174+
}
175+
if targetVersion < startVersion {
176+
return targetVersion, nil
177+
}
178+
if s.empty || s.lastVersion < targetVersion {
179+
return 0, fmt.Errorf("changelog ends below version %d: %s", targetVersion, s)
180+
}
181+
if s.firstVersion > startVersion {
182+
return 0, fmt.Errorf("changelog starts above version %d: %s", startVersion, s)
183+
}
184+
return targetVersion, nil
185+
}
186+
187+
func (s changelogSpan) String() string {
188+
if s.empty {
189+
return "the changelog is empty"
190+
}
191+
return fmt.Sprintf("the changelog holds versions %d to %d", s.firstVersion, s.lastVersion)
192+
}
193+
194+
// replayTreeChangeSets passes the changesets of treeName in versions startVersion to endVersion
195+
// to fn. It refuses an entry whose version does not match its index, and an upgrade of treeName.
196+
func replayTreeChangeSets(
197+
stream wal.ChangelogWAL,
198+
span changelogSpan,
199+
startVersion, endVersion int64,
200+
treeName string,
201+
fn func(version int64, changeSet proto.ChangeSet) error,
202+
) error {
203+
// #nosec G115 -- replayEnd placed both versions inside the span, so both indexes are positive
204+
startIndex, endIndex := uint64(startVersion-span.delta), uint64(endVersion-span.delta)
205+
return stream.Replay(startIndex, endIndex, func(index uint64, entry proto.ChangelogEntry) error {
206+
// #nosec G115 -- changelog indexes are far below MaxInt64
207+
if want := int64(index) + span.delta; entry.Version != want {
208+
return fmt.Errorf("changelog entry at index %d has version %d, want %d", index, entry.Version, want)
209+
}
210+
for _, upgrade := range entry.Upgrades {
211+
if upgrade.Name == treeName || upgrade.RenameFrom == treeName {
212+
return fmt.Errorf("changelog version %d upgrades tree %q (name=%q rename_from=%q delete=%t)",
213+
entry.Version, treeName, upgrade.Name, upgrade.RenameFrom, upgrade.Delete)
214+
}
215+
}
216+
for _, cs := range entry.Changesets {
217+
if cs.Name != treeName {
218+
continue
219+
}
220+
if err := fn(entry.Version, cs.Changeset); err != nil {
221+
return err
222+
}
223+
}
224+
return nil
225+
})
226+
}

0 commit comments

Comments
 (0)