Parallel backfill batches publish an oldest_ingest_ledger that claims un-ingested ledgers

[aristidesstaffieri]

Summary

Backfill advances oldest_ingest_ledger once per batch, from inside that batch's own transaction, while batches run in parallel. A batch that succeeds above a batch that failed still lowers the cursor. The published cursor then claims contiguous history across ranges that were never written.

Cause

1. Each batch's final flush passes its own start ledger as the cursor value — internal/services/ingest_backfill.go:367, applied at :283 (and :373/:384 when the ledger count is an exact multiple of the insert batch size).

2. UpdateMin is monotonically decreasing — internal/data/ingest_store.go:198-203:

   UPDATE ingest_store SET value = LEAST(value::bigint, $2)::text WHERE key = $1

3. Batches execute concurrently, so the final cursor is the minimum start ledger among successful batches. Failed batches return before their flush and contribute nothing. The result has no relationship to which ranges actually landed.

4. reconcile_oldest_cursor cannot repair this. It compares the stored value against MIN(ledger_number) and returns early unless actual_min > stored (internal/ingest/timescaledb.go:223). A gap above the minimum is invisible to it, and it only moves the cursor forward.

Proof

The repo's real UpdateMin statement and the verbatim reconcile_oldest_cursor function, run against Postgres 16. A [1000-1749] backfill as three parallel batches, middle batch succeeds:

batch A [1000-1249] FAILS   batch B [1250-1499] OK   batch C [1500-1749] FAILS

cursor after backfill "succeeds" (exit 0, see #688):  1250
actual data:                    min=1250  max=1999  rows=500
gap inside the claimed range:   1500-1749  (250 ledgers missing)
after reconcile_oldest_cursor:  1250   (unchanged)

Lowest batch succeeds instead, middle fails:

batch A [1000-1249] OK   batch B [1250-1499] FAILS   batch C [1500-1749] OK

cursor:                         1000
gap inside the claimed range:   1250-1499  (250 ledgers missing)
after reconcile_oldest_cursor:  1000   (unchanged)

Both cases also satisfy the integration assertion oldest <= expectedOldestLedger (internal/integrationtests/infrastructure/backfill_helpers.go:226), so CI passes.

Consequence

Protocol state is produced for ledgers that have no base rows. The history migration replays from the cursor using the ledger backend, not the transactions table (internal/services/protocol_migrate.go:361). It folds protocol state straight across the gap, so protocol tables describe ledgers whose transactions, operations, and state_changes rows do not exist.

A later repair of the base data does not repair the protocol data. This is the permanent part:

1. A partial backfill leaves the cursor at 1250 and exits 0.

2. protocol-migrate history reads the cursor via ResolveStartLedger (internal/services/protocol_migrate_history.go:90-98), replays from 1250, and sets the history migration status to StatusSuccess.

3. Someone re-runs the backfill for [1000-1249]. It succeeds. The cursor drops to 1000.

4. Re-running protocol-migrate history skips the protocol entirely — internal/services/protocol_migrate.go:207-209:

   if s.strategy.MigrationStatusField(p) == data.StatusSuccess {
       log.Ctx(ctx).Infof("Protocol %q %s migration already completed, skipping", pid, s.strategy.Label)
       continue
   }

The gate reads only the stored status, never the cursor. Protocol history for 1000-1249 is never produced, and no command will produce it.

Potential fix

Stop writing the cursor per batch. Collect the batch results, compute the longest contiguous successful prefix downward from the previous cursor value, and issue one UpdateMin for that prefix after every batch has settled. A batch that succeeded above a failed batch then contributes nothing to the cursor, which is correct: its rows exist, but history is not contiguous to it.

Two supporting changes matter as much:

• Make the migration status gate cursor-aware, or record the ledger the migration started from, so a cursor that later moves below it re-opens the work. Without this, the permanent gap stays reachable through any route that lowers the cursor.
• Fail the job on partial success (Backfill exits 0 when batches fail, so a partial backfill reports success #688), so step 1 above never looks clean.

Related

• Backfill exits 0 when batches fail, so a partial backfill reports success #688 — backfill exits 0 when batches fail. That issue hides this one.
• Not yet filed: on SIGTERM, batches queued but never started are recorded as successes (internal/services/ingest_backfill.go:188), so the failed-batch set is itself under-reported.