Repository navigation
Commit 53168c2
seidb: add changelog mode and --inspect-plan to speed up EVM digest (#4419)
The kvrepair drill-down runs `seidb evm-logical-digest` many times at
one height on a reserve and on a production node: one digest, then one
`--inspect-bucket --list` report for each bucket and key prefix. The
comparison height is almost never a memIAVL snapshot height, so each run
uses `--memiavl-open-mode replay`. That mode opens the full DB, catches
up every module through the changelog, and walks the EVM tree with
random reads, and it does this again for each report. On a mainnet node
the gap between snapshots is many thousands of blocks, so the drill-down
is slow.
This PR adds two flags and does not change any report value.
`--memiavl-open-mode changelog` scans the `kvs` file of the newest
snapshot at or below `--height` and merges in a sorted overlay of the
EVM writes from the changelog versions above it
(`memiavl.ReplayTreeChangelog`, `scanMemiavlChangelogEVMLeaves`). It
reports the same rows and the same mode labels as replay
(`semantic-replay`, `translator-replay`), so `kvrepair-export` accepts
its reports. It refuses a changelog that does not reach the height, that
starts above the snapshot, or that upgrades the `evm` tree in that
range. The stream opens the snapshot `kvs` file before the scan, so a
prune by the running node does not remove the data under it. When the
node prunes the selected snapshot before the stream opens, the error
wraps `memiavl.ErrSnapshotPruned`, and changelog mode selects the
snapshot again one time. `--inspect-plan` reads a JSON list of inspect
reports and takes all of them from one scan (`inspectFanout`); each
report is byte-equal to the report of a single run with the same flags.
It refuses two reports that write one file, by device and inode, so a
relative path, a symlink, or a hard link does not get past the check. To
make this possible, the memiavl and composite paths now go through one
leaf stream (`memiavlLeafStream`), account inspect lists come out in key
order (before, `--list-limit` kept a subset in map order), and
`entryHash` no longer allocates for account and storage rows.
The EVM logical tools (`evm-logical-digest`, `--inspect-plan`, and
`kvrepair-export`) move into a new package,
`sei-db/tools/cmd/seidb/operations/evmlogical`. The command names and
flags do not change. The FlatKV bucket names and the physical-key
classifier stay in `operations` and are exported, so both packages use
the same copy. The `evmlogical` tests have their own copy of some small
test-data helpers (`addrN`, `noncePair`, `newTestMemiavlStore`, and
others), because there is no shared test-helper package for
`operations`.
The first commit adds golden tests that pin the digest and inspect JSON
for every backend, normalization, and open mode on the current code. No
later commit changes an old golden case. The only output text that
changes is descriptive: the `normalization` field of semantic and
translator reports in replay mode now has the same text as in snapshot
mode ("raw memiavl EVM keys" in place of "replayed memiavl EVM keys"),
and changelog reports have their own `source` text. Review
`changelog_range.go` most closely, mainly the version and upgrade
checks. The tool is read-only and has no consensus or state impact.
Validation: ramtest (also with `-race`) for `sei-db/tools/...` and
`sei-db/state_db/sc/memiavl/...`, with tests that compare changelog mode
to replay mode at many heights, including the kvrepair export end-to-end
test in both modes. On a harbor chain (4 composite validators and 1
memIAVL reserve, snapshot at 20000), every changelog report was equal to
the replay report at heights 15000 to 28000: 74 pairs of digest, bucket
list, and shard reports over two runs. The reserve and composite digests
at 28000 were equal, and `kvrepair-export` accepted the changelog
reports. The EVM overlay was about 1 key for each block on that chain
(8,260 keys, 83 KB for 8,000 blocks). On two pacific-1 shadow nodes at
height 235215000, a memIAVL-only node 85,000 blocks above its snapshot
and a node with a complete migration, the changelog and replay reports
were equal on each node, the two nodes had the same final digest over
707,583,826 rows, and the `--inspect-plan` reports were byte-equal to
single runs. On the memIAVL node, the changelog digest took 405 s with a
24 GB peak RSS, and the replay digest took 2,286 s with a 142 GB peak
RSS. A narrow storage list took 120 s, and 10,052 s with replay.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>1 parent 84852aa commit 53168c2
19 files changed
Lines changed: 7472 additions & 448 deletions
File tree
- sei-db
- state_db/sc/memiavl
- tools/cmd/seidb
- operations
- evmlogical
- testdata
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
0 commit comments