test(index): stabilize lightweight 4-bit PQ recall

[BubbleCal]

Other Changes

The lightweight 4-bit IVF_PQ / IVF_HNSW_PQ matrix reduced PQ from 32 to 4 subvectors in #8733. With 32-dimensional inputs, that produces only a 2-byte code. The fixture seeds its input data but not IVF/PQ training, so the single-query R@10 result can fall below the required 0.5 threshold.

This change:

• uses 32 subvectors only for the lightweight 4-bit matrix, restoring its original 16-byte code capacity
• leaves the 8-bit matrix and focused M4 lifecycle, multivector, and bulk-distance coverage unchanged
• derives the metadata expectation from the actual PQ build parameters
• keeps recall >= 0.5, refine=4, ef=64, and all production code unchanged

This is a test-only alternative to #8767. That PR makes production KMeans/IVF/PQ training seedable and covers the broader #8764 scope, including multivector. This PR intentionally limits itself to restoring quality margin in the 4-bit matrices without adding a production API.

Measurements

Environment: Apple M4, 24 GiB RAM, macOS 26.6.1, Rust 1.97.0, prebuilt debug test binary, LANCE_CPU_THREADS=4, and --test-threads=1. Each configuration had 3 warm-ups. Per-metric results use 100 alternating M4/M32 runs; the complete matrix uses 30 alternating runs. Compilation time is excluded. Temporary measurement instrumentation was not committed.

Lower is better for runtime:

Scenario / metric	Baseline M4	This PR M32	Change
L2 case p50 wall time	87.1 ms	120.2 ms	1.38x slower (+33.1 ms)
Cosine case p50 wall time	88.7 ms	127.8 ms	1.44x slower (+39.1 ms)
Dot case p50 wall time	84.6 ms	124.7 ms	1.47x slower (+40.1 ms)
Three-case matrix p50 wall time	194.8 ms	304.6 ms	1.56x slower (+109.8 ms)

Higher is better for recall:

Metric	Baseline mean / min R@10	This PR mean / min R@10	Mean change	Passing runs
L2	0.712 / 0.4	0.997 / 0.9	+0.285 (+28.5 pp)	99/100 -> 100/100
Cosine	0.731 / 0.3	0.999 / 0.9	+0.268 (+26.8 pp)	99/100 -> 100/100
Dot	0.790 / 0.5	0.981 / 0.9	+0.191 (+19.1 pp)	100/100 -> 100/100

The equal-weight mean across all three metrics improved from 0.744 to 0.992 (+24.8 pp). The complete M4 matrix passed 29/30 runs; M32 passed 30/30.

Validation

• LANCE_CPU_THREADS=4 cargo test -p lance --lib pq_4bit -- --test-threads=1 (6 passed)
• focused 8-bit IVF_HNSW_PQ matrix (3 passed)
• alternating stability/recall protocol above (300/300 M32 per-case runs passed)
• cargo clippy --all --tests --benches -- -D warnings
• cargo fmt --all -- --check
• git diff --check

Related to #8764 and #8767.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Restoring 32 subvectors for the 4-bit fixtures gives the affected IVF-PQ and IVF-HNSW-PQ matrices adequate code capacity while retaining the recall threshold and sub-second runtime. This is independently useful test coverage; #8767 can continue separately as the broader deterministic-training change.