perf(fts): index residual compound rows once

[BubbleCal]

Performance issue

Partially indexed row-granularity FTS currently falls back to a separate flat residual scan for each leaf of a compound query. A Boolean, Boost, or same-field MultiMatch query can therefore read and tokenize the same small append-only tail repeatedly.

Change

Add a same-column hybrid path for bounded compound queries with small unindexed tails:

• scan and tokenize the residual rows once per query;
• retain query-term-only postings while collecting corpus and query-term statistics in the same pass;
• byte-rechunk residual input and build bounded query-local shards on the CPU pool;
• reuse the committed segment's already-loaded tokenizer assets;
• build the committed scorer while residual tokenization is running;
• search the committed postings and residual postings concurrently;
• merge both bounded top-k arms by score descending, then row ID ascending.

The scoring contract intentionally matches the existing partially indexed Row flat-search approximation:

• the indexed arm uses committed-index BM25 statistics;
• the residual arm starts with the committed statistics and adds residual document count, token count, and query-term document frequencies collected during its one required tokenization pass;
• exact-term matching, MUST_NOT exclusion, row-domain pruning, and deterministic tie ordering remain exact for supported query shapes;
• cross-arm scores and top-k may drift from a fully rebuilt index until indexing catches up.

This is not the fast_search contract: fast_search omits residual rows, while this path searches them approximately.

The planner fails closed to the established fallback for cross-column queries, Phrase or fuzzy leaves, filters, external row masks, deletions, fragment-scoped scans, list-element documents, unbounded queries, incomplete or overlapping physical coverage, rewritten/overlaid indexed sources, all-unindexed datasets, and residual tails above 100,000 physical rows. Empty analyzed term sets short-circuit without scanning the residual input.

Tests cover query-local term allowlisting and mixed corpus statistics, explicit row IDs, residual-only TF ordering, Boost, Boolean MUST/SHOULD/MUST_NOT, same-column MultiMatch, score ties, empty analyzed queries, bounded parallel shards, tokenizer reuse, fallback gates, and unsafe physical row-domain transitions.

Benchmark

Serialized current-head ABBA benchmark on one exclusive c4-highmem-16 VM (16 vCPU, 121 GiB RAM, us-central1-c) using release-with-debug and 8 workers. The frozen fixture contains 10,010,000 rows: 10,000,000 indexed rows plus 10,000 unindexed append-only rows.

Baseline: 338e100ca76e91ff593fa0d479245e1209489a8f
This PR: 7c30fa8d2296331576d1e6a10729f744bba62c55

Warm c1 uses one query at a time; warm c5 uses five frozen queries concurrently. Both cover must_should, should_sum, and boost_negative at k=10/100, with one prewarm, one warmup, three timed repetitions, and two ABBA trials per arm. Cold runs drop the page cache before every arm and report the mean of two first-query trials.

Scenario / metric	Baseline	This PR	Benefit
Warm c1 p50, 6 shape/k cells (lower is better)	1,650.156–1,727.691 ms	937.198–959.584 ms	1.7384–1.8312x speedup
Warm c1 throughput, 6 shape/k cells (higher is better)	0.576515–0.601026 q/s	1.047072–1.062625 q/s	1.7521–1.8162x throughput
Warm c5 p50, 6 shape/k cells (lower is better)	6,497.480–6,607.972 ms	3,194.601–3,212.246 ms	2.0334–2.0618x speedup
Warm c5 throughput, 6 shape/k cells (higher is better)	0.752992–0.766681 q/s	1.542985–1.550421 q/s	2.0209–2.0548x throughput
Cold first query, should_sum, k=10 p50 (lower is better)	1,935.677 ms	1,401.297 ms	1.3813x speedup
Cold first query, should_sum, k=100 p50 (lower is better)	1,942.472 ms	1,399.045 ms	1.3884x speedup

Activation preflight passed all 18 shape/query/k plans with HybridCompoundFtsScorer and without the flat fallback. Across 18 warm-c1, 90 warm-c5, and two cold ordered row+f32 digest cases, there was zero intra-build drift. This fixture also had zero observed cross-build digest changes, but that observation is not an exactness guarantee for arbitrary residual distributions.

Fixture provenance SHA256: dbd566d64c923b220d668718231d578a5ca075fe20708a3e4f9ecc40c9b7dcb4. Query manifest SHA256: 2efbbf779e213918c33d7517d069994a05a0b30cf9a9af4d22f9d0c9657d7dd1. Baseline module SHA256: a07439a03a0c7a5a367c3d3d37846283da528943bf3b0a189e4e9c37b2b33ee5. Target module SHA256: 56a455c8e1d9e2746f8a8da12ba7f495e4ca0e526b1906fc3cd2d550e1cec617. Harness SHA256: 454798e91c85ff3f98f4857ed02b79d69c67dca5c0f755feba24566347520a12. Analyzer SHA256: 12e2e83fbd3153e37088abd9bd3d6839713818946beffabfd7255c920442b3a7. Results fingerprint SHA256: 228daf0fd258821a1989ae2955be4a22e7e2f23811293376ad4edde1bfa328c2.

Validation

• cargo fmt --all -- --check
• git diff --check
• serialized current-head warm/cold ABBA benchmark and activation/determinism preflight described above
• GitHub CI for the current head
• no local Rust tests or clippy were run, as requested

[lance-gatekeeper]

❌ Gate recommendation: request changes.

1 fixed / 3 remain. The expanded physical-file identity now rejects cross-base replacements and base rebinding, but this head still does not compile and deferred remapping can still advance the only history watermark past stale physical postings.

Keep immutable physical-source provenance distinct from the remap watermark, and correct the iterator and path arguments so the focused regressions can build and exercise that unsafe transition.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The visibility adjustment keeps the query-local stats type confined through its crate-only re-export, and the affected crate check passes. The author intentionally accepts that cross-arm ranking may drift until indexing catches up; no further change is requested for that trade-off.