perf: assemble stable-row-id prefilter allow lists from cached per-fragment pieces

[JaySon-Huang]

Fixes #8849 (assembled-from-pieces variant; see the issue for the whole-dataset allow-list cost analysis)

Problem

DatasetPreFilter::do_create_deletion_mask_row_id builds a whole-dataset allow list by loading every fragment's RowIdSequence and deletion vector and folding them into one RowAddrTreeMap. The cost is proportional to the dataset size, not to what changed. The whole mask is cached under (manifest version, restrict_hash), which amortizes less than it looks:

• every commit bumps the version and invalidates the entry, so the first prefiltered query after any write reloads every fragment;
• distinct restrict_to sets get distinct cache entries, so a workload with varying fragment restrictions rebuilds repeatedly at the same version;
• the resident mask scales with the dataset.

Change

Assemble the allow list from per-fragment pieces. Each piece is one fragment's live stable row ids (row id sequence minus deletion vector, exactly what the old fold OR-ed in for that fragment), cached in the dataset metadata cache under its content identity:

• the row id generation (RowIdMeta), the same identity RowIdSequenceKey has used since RowIdSequenceKey cache key lacks a dataset version/generation — stale hit across drop + recreate at the same URI #7645 — deletes and added columns leave it untouched;
• the deletion file identity (read_version/id/file_type/base_id, mirroring DeletionFileKey).

Pieces are fetched concurrently and OR-ed on a blocking thread. The whole-mask cache under (version, restrict_hash) is kept, so repeated identical queries still hit it unchanged.

• Commits that leave a fragment's row ids and deletions untouched keep its piece warm: a post-commit rebuild only reloads touched fragments.
• Overlapping restrict_to sets share pieces instead of reloading them.
• Missing fragments (index bitmaps referencing removed fragments) contribute no piece, as before.

Correctness

A piece's content is fully determined by its key identity: any change to a fragment's row ids or deletions changes row_id_meta or the deletion file identity, so a stale piece cannot be served. mask(F) = OR(pieces(F ∩ manifest)) is identical to the old restricted fold (#6563/#6877 semantics preserved).

Tests: the existing prefilter mask tests (including the #6877 regression test) pass unchanged, plus a new test test_row_id_allow_list_pieces_invalidate_on_new_deletions covering cross-commit invalidation and restrict semantics. Full --lib suite: 3223 passed, 0 failed.

Performance

Local NVMe, 1M rows / 100 fragments / 0.5% scattered deletions / IVF_FLAT index, warm caches:

scenario	before	after
cold full allow-list build	4.4 ms	3.7–4.0 ms
repeated identical query (whole-mask cache)	~10 µs	~10 µs
rebuild after a commit deleting 1 of 100 fragments	3.5 ms (full rebuild)	348 µs
restrict set overlapping a previous one (50% shared)	1.9 ms	933 µs
any restrict set once all pieces are warm	1.9 ms	71 µs

On object storage the post-commit rebuild drops from ~N fragment loads (one round trip each) to only the touched fragments, which is where this mask cost dominates prefiltered searches on large tables.

Notes / follow-ups

• Keeping the whole-mask cache means a mask can be resident twice (pieces + composed). If that matters for memory-constrained setups, the composed entry could be dropped and rebuilt from warm pieces in ~71 µs at the 1M-row scale.
• A further step (not included) is intersecting the index-bitmap union with a filter's fragment coverage for selective scalar-index prefilters; FilterLoader currently does not expose fragment coverage.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The content-keyed per-fragment cache is a sound reuse boundary, but piece construction must preserve the existing async/CPU separation. Load row IDs and deletion vectors asynchronously, then run the clone, deletion masking, and tree construction for each miss on spawn_cpu; the final merge can remain on the CPU pool.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The code tree is unchanged from the prior reviewed revision, so the async/CPU boundary issue remains. The content-keyed per-fragment cache is a sound reuse boundary, but load row IDs and deletion vectors asynchronously, then run the clone, deletion masking, and tree construction for each miss on spawn_cpu; the final merge can remain on the CPU pool.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The prior async/CPU boundary issue is fixed: row-ID and deletion-vector loading remains asynchronous, while each piece’s clone, deletion masking, and tree construction now run on the CPU pool. The content-identity cache and restricted-mask semantics remain intact, and the focused cross-commit invalidation test passes.