feat(migrate): --rebuild — wipe and re-derive a protocol's indexed state - #692
Open
aditya1702 wants to merge 12 commits into
Open
feat(migrate): --rebuild — wipe and re-derive a protocol's indexed state#692aditya1702 wants to merge 12 commits into
aditya1702 wants to merge 12 commits into
Conversation
…otocol advisory lock Current-state migrations (and the upcoming rebuild mode) must not write the same protocol's tables concurrently. Each run try-locks a per-protocol advisory lock on a dedicated connection for its duration; a held lock fails the run before any status is marked. Live ingestion is deliberately not a party — the per-ledger cursor CAS already arbitrates which writer folds a given ledger. Plain history migration is append-only and does not contend.
…ry bounded backfill Current-state rebuild (engine.rebuild): after taking the per-protocol advisory lock, one transaction per protocol resets the migration cursor to start-ledger − 1 and deletes every current-state row via the processor's WipeCurrentState. The cursor UPDATE takes the row lock live ingestion's per-ledger CAS needs, so live serializes against the wipe and skips the protocol's folds until the migration hands ownership back at the frontier. validate() re-admits protocols whose migration already succeeded and refuses in_progress residue. History rebuild (bounded fold): history rows are per-ledger records with no running totals, so any range rebuilds in isolation. The service validates a COMPLETED history migration, clamps the range to [oldest retained ledger, committed history frontier], deletes the protocol's state_change_id-namespace rows in 10k-ledger slices (each its own transaction with the DML decompression cap lifted; to_id carries the ledger in its high 32 bits, so chunk skipping and compressed-batch minmax metadata prune every slice), then re-derives the range through the migrate pipeline with no cursor reads/writes, no live-frontier gating, and no CAS. Wipes never touch contract_tokens, protocol_wasms, or protocol_contracts — classification owns those and nothing rebuilds them.
current-state gains --rebuild; history gains --rebuild with optional --from-ledger/--to-ledger (valid only with --rebuild, defaulting to the full retained window). Help text states the destructiveness plainly. The history rebuild honors --oldest-ledger-cursor-name the same way the plain history migration does.
…y delete toid.AfterLedger is the maximum to_id within a ledger, not the first of the next one, so the upper bound must be inclusive. Also wrap the errors the wipe wrappers return.
Unit coverage: current-state rebuild re-admits succeeded protocols, wipes exactly once, and resets the cursor in the same transaction (a failed wipe rolls the reset back); in_progress residue refuses before any wipe; non-rebuild behavior unchanged. History rebuild: status/classification validation, range clamping to [oldest retained, lowest committed frontier], 10k-ledger delete slicing verified at every slice edge, bounded folds that never touch the cursor, and advisory-lock refusal. Data layer: namespace ∩ ledger-range deletes across all three namespaces, idempotent on retry. Integration: BlendMigrationTestSuite gains a phase that deletes one migrated BORROW state change, runs protocol-migrate history --rebuild over exactly that ledger, and asserts the row returns with other namespaces, contract_tokens, the cursor, and both migration statuses untouched.
The cursor name was configurable via --oldest-ledger-cursor-name on both ingest and protocol-migrate history, but nothing ever needed a non-default value, and a mismatch between the two commands silently breaks history migration's start-ledger resolution. The key is now the data.OldestLedgerCursorName constant everywhere, matching the hard-coded latest-ledger cursor.
…cenario The rebuild exists for rows that are present with wrong values, not just missing ones. The integration phase now corrupts the REPAY amount in place, deletes the BORROW row, rebuilds the multi-ledger range spanning both, and asserts the corrupted value is replaced by the re-derived one and the deleted row returns. Because state_change_ids are deterministic, the corrupted row's repair specifically witnesses the rebuild's delete step — a silently failing delete would leave the wrong amount in place.
The lock had only indirect coverage through the migration and rebuild Run paths. Pin the primitive itself: deterministic per-protocol IDs that differ across protocols, refuse-while-held with release restoring acquirability, no contention between distinct protocols, and — the cleanup path — a multi-protocol acquisition failing mid-list frees the locks it already took instead of wedging every protocol before the held one.
Every command's Run now takes its own strategy's per-protocol advisory lock (current-state scope or history scope) and the engine takes none — it is pure migrate machinery again, with no lock flag, no rebuild flag, and no wipe logic. History migrations gain mutual exclusion they previously lacked. The current-state rebuild moves out of the engine into its own service, mirroring the history rebuild: validate → lock → wipe → re-derive. Its wipe transaction resets the migration status to not_started alongside the cursor and the row deletes, which is what lets it reuse the engine's normal lifecycle unmodified. The current-state lock key bytes are unchanged; the history scope adds a new key.
The bounded fold shared only the shape of the engine loop, none of its machinery — cursor init, frontier gating, CAS, handoff, and per-window membership refresh are all live-race concerns a below-frontier replay never has. reDerive is a plain fetch/extract/process/persist loop over the inclusive range, loading membership once (classification for every ledger at or below the frontier committed before the run started). The engine loses its bounded mode and returns to exactly its pre-rebuild form.
…, mirroring current-state The history rebuild now shares the current-state rebuild's shape exactly: validate → lock → wipe → run the unmodified migration engine. Its wipe resets the protocol's cursor to the retention floor and the migration status to not_started (one transaction, committed before any delete so live ingestion stops writing the protocol's history and the deletes race nothing), then removes the protocol's rows over the retained window in ledger slices. The engine folds from the floor and hands off to live at the frontier, exactly as a first migration does. This drops --from-ledger/--to-ledger and the dedicated replay loop. Range scoping traded per-fix speed for symmetry; it returns if history migration is later folded into backfilling, where the range-scoped delete+re-derive is the natural shape.
aditya1702
requested review from
aristidesstaffieri
and
a balanced review from Copilot
August 21, 2026 14:00
Contributor
There was a problem hiding this comment.
Pull request overview
Adds destructive rebuild support for protocol current state and history while coordinating migrations with advisory locks.
Changes:
- Adds
--rebuildmigration modes with protocol-specific wipes. - Adds scoped advisory locking and sliced history deletion.
- Standardizes the oldest-ingest cursor and expands rebuild tests.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
cmd/ingest.go |
Removes configurable oldest-cursor flag. |
cmd/protocol_migrate.go |
Adds rebuild CLI options. |
internal/data/blend/wipe.go |
Wipes Blend current-state tables. |
internal/data/ingest_store_test.go |
Updates canonical cursor test keys. |
internal/data/sep41/wipe.go |
Wipes SEP-41 current state. |
internal/data/statechanges.go |
Adds namespace/range deletion. |
internal/data/statechanges_test.go |
Tests scoped history deletion. |
internal/ingest/ingest.go |
Uses the canonical oldest cursor. |
internal/ingest/timescaledb_test.go |
Updates cursor names in tests. |
internal/integrationtests/blend_test.go |
Tests Blend history repair. |
internal/services/blend/processor.go |
Exposes Blend current-state wipe. |
internal/services/ingest.go |
Removes cursor-name configuration. |
internal/services/ingest_backfill.go |
Uses the canonical cursor. |
internal/services/ingest_live.go |
Uses the canonical cursor. |
internal/services/ingest_live_test.go |
Updates ingestion setup. |
internal/services/ingest_test.go |
Updates ingestion fixtures and processors. |
internal/services/mocks.go |
Adds processor wipe mock. |
internal/services/protocol_migrate_current_state.go |
Locks current-state migrations. |
internal/services/protocol_migrate_current_state_rebuild.go |
Implements current-state rebuilds. |
internal/services/protocol_migrate_current_state_test.go |
Tests current-state locking. |
internal/services/protocol_migrate_history.go |
Locks history migrations. |
internal/services/protocol_migrate_history_rebuild.go |
Implements sliced history rebuilds. |
internal/services/protocol_migrate_history_test.go |
Tests history locking. |
internal/services/protocol_migrate_lock.go |
Implements scoped advisory locks. |
internal/services/protocol_migrate_lock_test.go |
Tests lock behavior. |
internal/services/protocol_migrate_rebuild_test.go |
Tests rebuild workflows. |
internal/services/protocol_migrate_test.go |
Extends the test processor. |
internal/services/protocol_processor.go |
Adds the wipe interface method. |
internal/services/sep41/processor.go |
Exposes SEP-41 current-state wipe. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+15
to
+16
| assert.Equal(t, migrateAdvisoryLockID(lockScopeCurrentState, "SEP41"), migrateAdvisoryLockID(lockScopeCurrentState, "SEP41"), | ||
| "the lock ID must be deterministic — every writer must derive the same key for a protocol") |
Comment on lines
+15
to
+19
| for _, table := range []string{ | ||
| "blend_pools", | ||
| "blend_positions", | ||
| "blend_reserves", | ||
| "blend_backstop_positions", |
| return fmt.Errorf("resetting history migration state for %s: %w", protocolID, txErr) | ||
| } | ||
|
|
||
| base := s.engine.processors[protocolID].StateChangeOrdinalBase() |
Comment on lines
+14
to
+15
| for _, table := range []string{"sep41_balances", "sep41_allowances"} { | ||
| if _, err := dbTx.Exec(ctx, "DELETE FROM "+table); err != nil { |
Comment on lines
289
to
+290
| func(cmd *cobra.Command, opts *migrationCommandOpts) { | ||
| cmd.Flags().StringVar(&oldestLedgerCursorName, "oldest-ledger-cursor-name", data.OldestLedgerCursorName, "Name of the oldest ledger cursor in the ingest store. Must match the value used by the ingest service.") | ||
| cmd.Flags().BoolVar(&rebuild, "rebuild", false, "Delete the protocol's history rows and rebuild them from the oldest retained ledger. Destructive: the protocol's history serves empty/partial data until the rebuild reaches the live frontier.") |
Contributor
|
We should update the migration documentation to include the parts needed for the rebuild, like the wipe logic. |
| "blend_oracle_prices", | ||
| "blend_auctions", | ||
| } { | ||
| if _, err := dbTx.Exec(ctx, "DELETE FROM "+table); err != nil { |
Contributor
There was a problem hiding this comment.
any reason not to just truncate here?
Since the update at the beginning of the transaction in protocol_migrate_current_state_rebuild.go takes a lock, it blocks ingestion for all protocols until the delete completes which could be expensive for some protocols and is unbounded, blocking indefinitely.
|
|
||
| // retainedWindow returns the wipe bounds: oldest retained ledger through | ||
| // live ingestion's committed tip. | ||
| func (s *protocolHistoryRebuildService) retainedWindow(ctx context.Context) (uint32, uint32, error) { |
Contributor
There was a problem hiding this comment.
can this also check start > latest break or oldest <= latest to avoid operator mistakes that throw this into a loop?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Two commands that fix a protocol's corrupted data after a bug fix ships:
Each command wipes the protocol's rows, then re-runs the normal migration. One invocation, no separate wipe step. Live ingestion keeps running.
Why wipe + remigrate
Columns like Blend cost basis and SEP-41 balances are running totals. The chain does not store them anywhere. The only fix that works for every protocol is replaying all events from the start.
Rejected alternatives:
state_changes— retention drops old rows, and a decode bug corrupts those rows too.How it works
Both rebuilds have the same shape: validate → lock → wipe → run the normal migration. The migrate engine itself is untouched by this PR.
Current-state rebuild:
in_progress.--start-ledger. It hands off to live ingestion at the tip, like any migration.History rebuild:
in_progress.Safety
contract_tokens,protocol_wasms,protocol_contractsare never touched. Nothing can rebuild them.Locks
One advisory lock per protocol per strategy. Only one current-state run and one history run can touch a protocol at a time. History migrations gain mutual exclusion they didn't have before.
Trade-offs (accepted on purpose)
Testing
in_progressrefusal, delete slice edges, lock behavior (including a failed multi-protocol acquire freeing its locks).history --rebuild. Both come back correct; other namespaces,contract_tokens, and statuses are untouched.make checkgreen.After merge: run both rebuilds for SEP41 in dev and confirm the known bad balance (CAZXR…EXSND) is fixed.
🤖 Generated with Claude Code