BUGFIX: per-scan TupleDesc/slot leak and per-rescan state leak

[dylon]

Release scan resources deterministically: fix per-scan TupleDesc/slot leak and per-rescan state leak

Fixes #211.

Summary

Two leaks on the diskann scan (rescore) path let a backend grow without bound.
The first is the TupleDesc reference leak reported in #211 — it is not a cosmetic
warning; it is accompanied by a leaked TupleTableSlot on every dead/invisible
rescored candidate, and on a table with ongoing UPDATE/DELETE traffic it grows
until the host runs out of memory.

1. TupleTableSlot + rowtype TupleDesc leaked when a heap pointer has no visible tuple

TableSlot::from_index_heap_pointer (src/util/table_slot.rs) creates the slot with
table_slot_create, which goes through MakeTupleTableSlot → PinTupleDesc(rd_att)
and so registers a descriptor pin on the current ResourceOwner. That pin — and the
slot itself — are released only by TableSlot's Drop
(ExecDropSingleTupleTableSlot).

The wrapper was constructed only on the success path:

let slot = PgBox::from_pg(pg_sys::table_slot_create(heap_rel.as_ptr(), null_mut()));
// ... index_fetch_tuple ...
if !valid {
    return None;          // <-- slot never wrapped: slot + TupleDesc pin leak
}
Some(Self { slot })

PgBox::from_pg is non-owning, so its Drop is a no-op — the early return leaks both.
!valid is hit whenever index_fetch_tuple finds no snapshot-visible version in the
HOT chain (deleted, updated-away, or not-yet-vacuumed TIDs), which the scan explicitly
expects. It is reached from the rescore path — get_full_distance_for_resort in both
sbq/storage.rs and plain/storage.rs, driven by next_with_resort — and SBQ always
rescores, with resort on by default. So on a churned table this leaks once per
rescored candidate, and the descriptor pin surfaces as the resource was not closed: TupleDesc ... (<oid>,-1) warning from #211.

Introduced by b480530 ("Fix HOT update handling"), which added the !valid early
return; before that the slot was always wrapped.

Fix: arm the RAII wrapper immediately after table_slot_create and fetch through
it, so every exit path — the early return and any unwind — drops the slot and releases
the descriptor. No caller changes: both callers already handle None.

2. Per-rescan StorageState accumulation, and amendscan freeing nothing

TSVScanState::initialize published the scan's StorageState with
PgMemoryContexts::CurrentMemoryContext.leak_and_drop_on_delete(...), i.e. tied to the
executor's per-query context. amrescan calls initialize again for every rescan
and overwrote the pointer without dropping the previous state, so a nested-loop or
lateral join orphaned one response iterator (plus a cloned SbqQuantizer) per outer
row, all held until the query ended. amendscan's entire body was gated behind
min_level <= DEBUG1 and freed nothing, so even a single scan's state lived until
per-query teardown.

Fix: make the state genuinely owned — Box::into_raw in initialize after
releasing the previous one, a release_storage() helper that nulls as it frees (so it
is idempotent), an unconditional release in amendscan, and a Drop impl on
TSVScanState as the backstop for scans torn down without amendscan (e.g. an aborted
query). The field stays a raw pointer so the existing call sites, which need
&mut TSVScanState alongside the state, are unchanged. amendscan's debug-stats block
now tolerates a scan that was never rescanned instead of expect-ing storage.

Impact (why #211 is not benign)

Observed on PostgreSQL 18.4 with pgvectorscale 0.9.0 serving a ~740k-row,
1024-dimension table under continuous re-indexing (so a steady supply of dead tuples),
with dense ANN queries plus lateral k-NN batch jobs:

• The resource was not closed: TupleDesc ... (17176,-1) warning fired at roughly
  650/min, one per rescored dead candidate.
• Six pooled backends grew to 7.6, 9.3, 13.4, 14.9, 17.4 and 19.5 GB — about
  82 GB — and the host OOM-killed. Backends that were idle still held multiple GB,
  i.e. the growth was retained, not query working set.
• Cluster settings ruled out ordinary query memory: work_mem = 4MB,
  maintenance_work_mem = 64MB.
• Dropping the diskann index (reverting that workload to an HNSW index) removed the
  growth completely: the same workload then held ~1.4 GB across all backends and stayed
  flat.

Minimal reproduction

PostgreSQL's own ResourceOwner reports each unreleased descriptor pin, so the leak is
directly observable without driving a backend to OOM. This takes a few hundred rows,
one query, and a couple of seconds:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS vectorscale;

CREATE TABLE leak_mve (id int primary key, embedding vector(3));
INSERT INTO leak_mve
SELECT g, ('[' || g || ',' || (g % 7) || ',' || (g % 5) || ']')::vector
  FROM generate_series(1, 500) g;
CREATE INDEX leak_mve_diskann ON leak_mve USING diskann (embedding);

-- Make most index entries point at tuples that are no longer visible. No VACUUM on
-- purpose: the index still references those dead TIDs, which is the steady state of
-- any table with ongoing UPDATE/DELETE traffic.
DELETE FROM leak_mve WHERE id % 2 = 0;

SET enable_seqscan = off;      -- ensure the index scan (and so the rescore) runs
SET client_min_messages = warning;

SELECT id FROM leak_mve ORDER BY embedding <=> '[1,2,3]'::vector LIMIT 10;

On PostgreSQL 18.4 with pgvectorscale 0.9.0 (57c88b7), that single query prints:

WARNING:  resource was not closed: TupleDesc 0x7fc0f20d65c8 (16796,-1)
...                                                    (x60)

60 slots and descriptor pins allocated and never freed by one small query. 16796
is leak_mve's rowtype OID, and each line is one TupleTableSlot that
table_slot_create allocated and pinned but nothing ever dropped. Nothing here is
reclaimed until the ResourceOwner is torn down, and each is charged to the backend
until then — which is why a busy table walks a backend up into the tens of GB.

Build	Dead tuples present	Leak warnings
0.9.0 (unpatched)	yes	60
this PR	yes	0
0.9.0 (control)	no — every tuple live	0

The control run matters: with no dead tuples the rescore never takes the !valid
branch and nothing leaks, which localises the defect precisely to that early return.

The same leak measured as backend memory

Scaling the reproduction up — 4 000 rows, half of them deleted, then ~8 000 index
rescans driven from a lateral join inside a single backend — shows the leaked
slots accumulating as resident memory. Same server, same query, only the extension
differs:

elapsed	unpatched backend RSS	patched backend RSS
3 s	182.6 MB	36.0 MB
9 s	328.2 MB	36.0 MB
15 s	388.0 MB	36.0 MB
21 s	454.8 MB	36.0 MB
24 s	493.6 MB (still climbing)	36.0 MB

Unpatched, the backend grows monotonically for as long as the workload runs and never
gives the memory back; patched, it is flat. Extrapolate that to a production table
under continuous churn and it is the 82 GB OOM described above.

Verification

• The regression test fails without the fix. Reverting only the RAII change and
  re-running table_slot_releases_tupledesc_on_dead_tuple gives:

  assertion `left == right` failed: the rowtype TupleDesc pin must be released
  when there is no visible tuple (leak: issue #211)
    left: 2     <- descriptor refcount after the call
   right: 1     <- refcount before it
  

  With the fix the count is conserved and the test passes.
• Full suite green: cargo pgrx test --no-default-features --features 'pg_test pg18 build_parallel' -- pg18
  → 84 passed; 0 failed; 10 ignored on PostgreSQL 18.4 (a --enable-cassert
  build), including both new tests.
• cargo fmt --check clean.

Tests

• util::table_slot::tests::table_slot_releases_tupledesc_on_dead_tuple — inserts a
  row, records its TID, deletes it, and calls from_index_heap_pointer with a snapshot
  that cannot see it. Asserts the call returns None and that the relation's
  rowtype TupleDesc reference count is unchanged. Fails before the fix (the count is
  left incremented), passes after.
• access_method::scan::tests::diskann_rescan_releases_state_each_iteration — a
  lateral join that rescans the index once per outer row over a deliberately churned
  heap. Covers the rescan release path and would crash on a double free or
  use-after-free rather than merely leak.

Existing accuracy/scan suites are unchanged and still pass.