Fix lookup-table reaper deleting rows referenced by concurrent ingestion - #222

What

The lookup-table reaper could delete a history lookup row (history_accounts, history_assets, history_claimable_balances, history_liquidity_pools) while live ingestion was concurrently inserting a reference to it, leaving a dangling reference in the history tables (history_operation_participants, history_effects, history_trades, …). The lookup row's id is the join key for an account's history, so at the API level this surfaces as:

• GET /accounts/{address} → 200 with the correct balance (the account looks healthy), but
• GET /accounts/{address}/payments (and /operations, /effects, /trades) → 404, because the join to history_accounts returns no rows.

Once the address transacts again, the loader re-inserts it with a new id, so those endpoints return 200 again but silently omit every earlier record. The operation rows themselves are never deleted — only the join key — so a reingest of the retained range recovers them.

Root cause

constructDeleteLookupTableRowsQuery built the reaper's delete as a single statement that both took a FOR UPDATE row lock and ran the NOT EXISTS orphan check:

WITH ha_batch AS (SELECT id FROM history_accounts WHERE id IN (...) ORDER BY id ASC FOR UPDATE)
DELETE FROM history_accounts WHERE id IN (
  SELECT e1.id FROM ha_batch e1 WHERE NOT EXISTS (... references ...))

The FOR UPDATE lock was intended to make the reaper block until any concurrent ingestion (which locks the same rows FOR KEY SHARE) commits. It does block — but under the default READ COMMITTED isolation the statement's snapshot is fixed when the statement starts, before it blocks. When the ingestion transaction it blocked on commits, the NOT EXISTS sub-queries still evaluate against the pre-commit snapshot, don't see the just-inserted references, report the row as orphaned, and delete it. Blocking on the lock does not refresh the snapshot the check runs under.

Fix

Split the lock and the orphan check into two separate statements within the reaper's transaction:

1. SELECT id FROM <table> WHERE id IN (...) ORDER BY id ASC FOR UPDATE — take the locks and wait out any in-flight ingestion.
2. DELETE FROM <table> WHERE id IN (...) AND NOT EXISTS (... references ...) — a separate statement, so its snapshot is taken after the lock wait resolves and it observes the references committed by the ingestion transaction the reaper blocked on. The FOR UPDATE locks from statement 1 are held for the rest of the transaction, so no new ingestion can insert a reference in between.

Scope

This fixes concurrency with live ingestion, which is where the reaper runs concurrently by design. Reaping concurrently with db reingest range (the non---force path, which uses ConcurrentInserts and does not take the FOR KEY SHARE lock on the parent) is a separate, narrower window and is left as a follow-up.

Tests

• TestReapDoesNotDeleteConcurrentlyReferencedRows (new): drives the exact interleaving — ingestion locks the account FOR KEY SHARE and inserts a referencing history_operation_participants row, uncommitted; the reaper blocks on the lock; ingestion commits; the reaper must leave the row in place. This test fails against the previous single-statement query (deletes the row) and passes with the fix.
• Updated TestConstructDeleteLookupTableRowsQuery and added TestConstructLockLookupTableRowsQuery for the new query builders.

🤖 Generated with Claude Code