Skip to content

Commit 792ff53

Browse files
Apply compiled KV repair files at a fixed height and generate them from digest inspect lists (#4400)
## Summary The app can now apply reviewed key/value repairs to module stores at a fixed height, and `seidb kvrepair-export` generates the repair file from two `evm-logical-digest` inspect lists. Each repair registers a handler with the existing `HardForkManager`, so it runs in `BeginBlock` at that height, with no governance proposal. A repair file is a consensus-breaking change: every validator must run a binary that holds the file before its height, and a node without the file stops at the next block. The directory `app/upgrades/kvrepair/repairs/` is empty, so this PR registers no handler and does not change behaviour. The file format and the generation steps are in `app/upgrades/kvrepair/repairs/README.md`. The intended use is on a stopped chain. The chain stops at `H`. The operator lists the keys in each bucket that the digest reports as different, on a memiavl reserve and on a production node at `H`. `kvrepair-export` writes a file with `target_repair_height` `H+1` and `state_height` `H`. The nodes restart with the new binary and continue in composite mode, with the EVM migration still in progress. - `sei-db/common/kvrepair/repair.go`: the file format. `Parse` refuses unknown fields, a field name used twice in one object, a field name that is not lowercase ASCII (encoding/json matches names without regard to case), data after the JSON value, bad hex, a missing `new`, a `null` old, and an entry with both `old` and `old_absent`. `Validate` refuses an empty name or chain ID, a `target_repair_height` that is not positive, a `state_height` that is not positive or not below `target_repair_height`, an empty entry list, an unknown store, an empty key, and a key twice in one repair. An entry without an old value is allowed only when `state_height` is `target_repair_height - 1`. `Encode` writes one entry per line, so a review diff shows one line for each key. - `sei-db/common/kvrepair/values.go`: `ValuesEqual` and `ReadsAsAbsent`. In the `evm` store, an all-zero storage slot, nonce, code hash, or balance, and an empty code value, read as absent. Every other value compares exactly, so `""` differs from an absent key, and code made of zero bytes is real code. - `app/upgrades/kvrepair/kvrepair.go`: `Register` loads every embedded file, validates it against the app's store keys, refuses a name used twice and a key that two repairs touch at one height, and registers one handler for each repair. The hard fork manager skips repairs for other chains. An invalid file panics at start. - `app/upgrades/kvrepair/kvrepair.go` (`Handler.ExecuteHandler`): for each entry, the handler checks the current value, writes `new` (or deletes the key), and reads it back. An entry whose key already holds `new` skips its old value check and is still written, so a node with correct state commits the same changeset as a repaired node. Any other entry must find its old value (`old`, or no value with `old_absent`). These comparisons use `ValuesEqual`, because a FlatKV account row returns a zero nonce for an account whose nonce was deleted while another field remains, where memiavl returns no value. A failed old value check or read-back returns an error, and the hard fork manager then panics in `BeginBlock`, so the node stops at that height and does not commit the block. - `app/app.go`: calls `kvrepair.Register` after the hard fork manager is created. - `sei-db/tools/cmd/seidb/operations/kvrepair_export.go`: `seidb kvrepair-export --reserve --prod --chain-id --name --target-repair-height` reads two inspect list reports and writes a repair file. It refuses a reserve that is not memiavl or whose mode is not `semantic` or `semantic-replay` (a translator report shares the migration mapping), a production report that is not composite or FlatKV, two reports that differ in height, bucket, offset, or prefix, a report that is not a list or that `--list-limit` cut off, an unknown bucket, a non-EVM physical key, a key listed twice, a target repair height that is not above the report height, and reports with no differences. For each key whose rows differ, `new` is the reserve value (`null` when it reads as absent) and `old` is the production value (`old_absent` when it reads as absent), so every entry has an old value. An all-zero row that only one side holds is deleted. An account row is split into the balance, nonce, and code hash fields that differ. Before it writes, the command encodes, parses, and validates the file as a node would. - `sei-db/tools/cmd/seidb/main.go`: registers `kvrepair-export`. - `sei-db/tools/cmd/seidb/operations/evm_logical_digest.go`: the inspect usage example gains the `--key-offset 4` that its key prefix needs. `.dockerignore` removes `*.md`, so the directory is embedded with `//go:embed all:` and holds a `.gitkeep`. ## Test plan - `sei-db/common/kvrepair/repair_test.go`: every invalid form listed above, including a second object or trailing garbage, a field twice in an entry or at the top, a field name in another case or with a rune that folds to a lowercase letter, and an entry without an old value across a gap. A repair with a gap loads when every entry has an old value. `""` and `"0x"` load as empty values, distinct from `null`. `Encode` writes one entry per line, omits an empty source, and round-trips through `Parse`. - `sei-db/common/kvrepair/values_test.go`: zero and absent compare as equal for each EVM field kind. Empty code reads as absent and zero-byte code does not. Misc keys, malformed storage keys, and non-EVM stores compare exactly. - `app/upgrades/kvrepair/kvrepair_test.go`: the embedded directory loads, and the loader refuses a name used twice and two repairs on one key at one height. - The handler sets and deletes keys at the target height. An entry that already holds `new` is written, and its old value check is skipped, as a store trace shows. - The handler fails and leaves the store unchanged for a current value that differs from `old`, a present key with `old_absent`, and a missing key with `old`. With no old value, the entry writes over any value. - A zero storage slot counts as holding a delete, and a zero nonce matches `old_absent`. A different non-zero value still fails. Misc and non-EVM keys compare exactly. - `sei-db/tools/cmd/seidb/operations/kvrepair_export_test.go`: storage sets, deletes, and creates. A zero row that only production holds is deleted. Code and misc keys map to their store keys. An account entry writes only the changed field. An account that only the reserve holds gets entries for its non-zero fields, and an all-zero account row that only production holds is deleted field by field. The command refuses each bad input listed above, and the command writes a file that loads. An end-to-end test inspects a real memiavl reserve and a real FlatKV store, exports every bucket, applies the entries to the FlatKV store, and inspects again: the new reports list no differences. - `sei-db/state_db/sc/composite/store_repair_migration_test.go`: one block during the EVM migration writes keys that are already in FlatKV, keys still in memiavl, keys that the same block's migration batch moves, and an account whose nonce is still in memiavl while its balance goes to FlatKV. Each key reads back the written value after the block and after the migration completes, and the FlatKV LtHash verifies. - Manual, stopped chain, on a harbor chain with 4 composite validators and a memIAVL-only reserve (fixture branch `yiren/a8-kvrepair-fixture-v2`, not for merge). A fixture seeds EVM state at 30, starts the migration at 60, and damages state on the validators at 300 while the migration is in progress: two storage slots set, two deleted, one added, a contract's code replaced, one nonce changed, and one nonce set to zero. The chain stoped with `stopped-height` and stopped at 803 with the migration in progress. `kvrepair-export` on inspect lists from the reserve and a validator at 803 wrote 11 entries in 4 files (account, code, storage, misc), including the zero-storage cleanup checkpoint (`0x1e`), which the storage damage also moved. The migration markers were left out. With the files compiled in and the repair at 804: - Every validator logged `repaired=11 already_correct=0`, and the reserve logged `repaired=0 already_correct=11`. - The `changeset` differed between a validator and the reserve at 471 heights from 300 to 803, and matched at every height from 804 to 990. Two validators had identical hash logs from 1 to 990. - The migration continued from its boundary and completed at about 1070. - At 1200, the EVM logical digest of a validator (FlatKV) and the reserve (memIAVL) matched in every bucket, count and digest. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent e8ddc72 commit 792ff53

14 files changed

Lines changed: 2007 additions & 5 deletions

File tree

‎app/app.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ import (
102102
appparams "github.com/sei-protocol/sei-chain/app/params"
103103
"github.com/sei-protocol/sei-chain/app/retiredoracle"
104104
"github.com/sei-protocol/sei-chain/app/upgrades"
105+
"github.com/sei-protocol/sei-chain/app/upgrades/kvrepair"
105106
v0upgrade "github.com/sei-protocol/sei-chain/app/upgrades/v0"
106107
"github.com/sei-protocol/sei-chain/evmrpc"
107108
evmrpcconfig "github.com/sei-protocol/sei-chain/evmrpc/config"
@@ -989,6 +990,7 @@ func New(
989990
// example: app.HardForkManager.RegisterHandler(myHandler)
990991
app.HardForkManager = upgrades.NewHardForkManager(app.ChainID)
991992
app.HardForkManager.RegisterHandler(v0upgrade.NewHardForkUpgradeHandler(100_000, upgrades.ChainIDSeiHardForkTest, app.WasmKeeper))
993+
kvrepair.Register(app.HardForkManager, keys)
992994

993995
app.RegisterDeliverTxHook(app.AddCosmosEventsToEVMReceiptIfApplicable)
994996

‎app/upgrades/kvrepair/kvrepair.go‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
// Package kvrepair applies reviewed key/value repairs to module stores as hard
2+
// fork handlers. Each repair is a JSON file under repairs/ that names one chain,
3+
// one height, and the entries to write or delete at that height.
4+
package kvrepair
5+
6+
import (
7+
"embed"
8+
"fmt"
9+
"io/fs"
10+
"path"
11+
12+
"github.com/sei-protocol/seilog"
13+
14+
"github.com/sei-protocol/sei-chain/app/upgrades"
15+
sdk "github.com/sei-protocol/sei-chain/sei-cosmos/types"
16+
"github.com/sei-protocol/sei-chain/sei-db/common/kvrepair"
17+
)
18+
19+
var logger = seilog.NewLogger("app", "upgrades", "kvrepair")
20+
21+
// repairsDir holds the repair files compiled into the binary.
22+
const repairsDir = "repairs"
23+
24+
//go:embed all:repairs
25+
var embeddedRepairs embed.FS
26+
27+
// Register validates every embedded repair against the store keys and
28+
// registers one handler for each. It panics on an invalid repair file.
29+
func Register(m *upgrades.HardForkManager, keys map[string]*sdk.KVStoreKey) {
30+
repairs, err := Load(embeddedRepairs, keys)
31+
if err != nil {
32+
panic(fmt.Errorf("kvrepair: %w", err))
33+
}
34+
for _, r := range repairs {
35+
m.RegisterHandler(NewHandler(r, keys))
36+
}
37+
}
38+
39+
// Load reads and validates every .json file under repairs/ in fsys. It also
40+
// refuses a repair name used twice, and a key that two files repair at the
41+
// same height.
42+
func Load(fsys fs.FS, keys map[string]*sdk.KVStoreKey) ([]kvrepair.Repair, error) {
43+
files, err := fs.ReadDir(fsys, repairsDir)
44+
if err != nil {
45+
return nil, err
46+
}
47+
storeExists := func(name string) bool {
48+
_, ok := keys[name]
49+
return ok
50+
}
51+
var repairs []kvrepair.Repair
52+
names := map[string]string{}
53+
targets := map[string]string{}
54+
for _, file := range files {
55+
if file.IsDir() || path.Ext(file.Name()) != ".json" {
56+
continue
57+
}
58+
filePath := path.Join(repairsDir, file.Name())
59+
data, err := fs.ReadFile(fsys, filePath)
60+
if err != nil {
61+
return nil, err
62+
}
63+
r, err := kvrepair.Parse(data)
64+
if err != nil {
65+
return nil, fmt.Errorf("%s: %w", filePath, err)
66+
}
67+
if err := r.Validate(storeExists); err != nil {
68+
return nil, fmt.Errorf("%s: %w", filePath, err)
69+
}
70+
if other, ok := names[r.Name]; ok {
71+
return nil, fmt.Errorf("%s: repair name %q already used by %s", filePath, r.Name, other)
72+
}
73+
names[r.Name] = filePath
74+
for _, e := range r.Entries {
75+
target := fmt.Sprintf("%s/%d/%s/%x", r.ChainID, r.TargetRepairHeight, e.Store, []byte(e.Key))
76+
if other, ok := targets[target]; ok {
77+
return nil, fmt.Errorf("%s: store %s key %x at height %d is also repaired by %s",
78+
filePath, e.Store, []byte(e.Key), r.TargetRepairHeight, other)
79+
}
80+
targets[target] = filePath
81+
}
82+
repairs = append(repairs, r)
83+
}
84+
return repairs, nil
85+
}
86+
87+
// Handler implements upgrades.HardForkHandler for one Repair.
88+
type Handler struct {
89+
repair kvrepair.Repair
90+
keys map[string]*sdk.KVStoreKey
91+
}
92+
93+
// NewHandler returns the hard fork handler that applies r.
94+
func NewHandler(r kvrepair.Repair, keys map[string]*sdk.KVStoreKey) upgrades.HardForkHandler {
95+
return Handler{repair: r, keys: keys}
96+
}
97+
98+
func (h Handler) GetName() string { return "kvrepair-" + h.repair.Name }
99+
100+
func (h Handler) GetTargetChainID() string { return h.repair.ChainID }
101+
102+
func (h Handler) GetTargetHeight() int64 { return h.repair.TargetRepairHeight }
103+
104+
// ExecuteHandler writes every entry and reads each one back. It returns an
105+
// error when a key that does not hold its new value holds something other than
106+
// its old value, or when a read-back does not match the write. Values compare
107+
// with kvrepair.ValuesEqual.
108+
func (h Handler) ExecuteHandler(ctx sdk.Context) error {
109+
var repaired, alreadyCorrect int
110+
for i, e := range h.repair.Entries {
111+
store := ctx.KVStore(h.keys[e.Store])
112+
current := store.Get(e.Key)
113+
if holdsTarget(e, current) {
114+
alreadyCorrect++
115+
} else if err := checkOld(e, current); err != nil {
116+
return fmt.Errorf("kvrepair %s: entry %d (store %s key %x): %w",
117+
h.repair.Name, i, e.Store, []byte(e.Key), err)
118+
} else {
119+
repaired++
120+
}
121+
// Entries that already hold their target are written too, so that a
122+
// correct node and a repaired node commit the same changeset.
123+
if e.New == nil {
124+
store.Delete(e.Key)
125+
} else {
126+
store.Set(e.Key, *e.New)
127+
}
128+
if got := store.Get(e.Key); !holdsTarget(e, got) {
129+
return fmt.Errorf("kvrepair %s: entry %d (store %s key %x): read back %x after the write",
130+
h.repair.Name, i, e.Store, []byte(e.Key), got)
131+
}
132+
}
133+
logger.Info("applied kv repair",
134+
"repair", h.repair.Name,
135+
"height", ctx.BlockHeight(),
136+
"repaired", repaired,
137+
"already_correct", alreadyCorrect,
138+
)
139+
return nil
140+
}
141+
142+
// holdsTarget reports whether current reads the same as the entry's new value.
143+
func holdsTarget(e kvrepair.Entry, current []byte) bool {
144+
// Not bytes.Equal: a FlatKV account row returns a zero nonce for an account
145+
// whose nonce was deleted while another field remains, where memiavl returns
146+
// nil.
147+
return kvrepair.ValuesEqual(e.Store, e.Key, current, optional(e.New))
148+
}
149+
150+
func checkOld(e kvrepair.Entry, current []byte) error {
151+
switch {
152+
case e.OldAbsent && !kvrepair.ReadsAsAbsent(e.Store, e.Key, current):
153+
return fmt.Errorf("expected the key to be absent, found %x", current)
154+
case e.Old != nil && !kvrepair.ValuesEqual(e.Store, e.Key, current, *e.Old):
155+
return fmt.Errorf("expected old value %x, found %x", []byte(*e.Old), current)
156+
}
157+
return nil
158+
}
159+
160+
// optional returns the bytes of v, or nil when v is nil.
161+
func optional(v *kvrepair.HexBytes) []byte {
162+
if v == nil {
163+
return nil
164+
}
165+
return *v
166+
}

0 commit comments

Comments
 (0)