fix(index): keep an index this build cannot read out of the erase path

[wombatu-kun]

Problem

retain_supported_indices filters the index list inside load_indices, and the commit path rebuilds the next manifest from exactly that list (build_manifest starts with let mut final_indices = current_indices;). For an index whose version this build cannot read, that turns "ignore it" into "delete it" on the next commit of any kind - an append, a delete, a config change. The dataset loses an index a newer Lance wrote and could still use.

Invisible inside one process: commit_transaction seeds the index cache with the unfiltered list, so only a cold reader sees the loss.

Fix

Split the two questions load_indices() was answering. load_all_indices returns every index the manifest names; DatasetIndexExt::load_indices applies the version filter on the way out. Readers see only what they can read; everything that decides what the next manifest says sees all of it:

• the commit path, so the manifest is rebuilt from the full list;
• name reservation, replace/removal selection, drop_index, the alter_columns cast guard;
• the planners - plan_compaction, optimize_indices, Dataset::validate - which look like readers but produce the next manifest's index list.

Compaction needs one rule beyond the split. Without stable row ids a rewrite moves every row address, and putting an index back in step means opening it, which this build cannot do: DefaultCompactionPlanner holds back the fragments an unreadable index covers so the rest of the table still compacts, and commit_compaction refuses a plan that rewrites them anyway - the boundary a custom planner, a hand-built CompactionPlan and a distributed driver all pass through.

migrate_indices skips an index it cannot open, and optimize_indices skips a name whose segments it cannot all read; both would otherwise have to open it.

IndexMetadataKey's CacheKeySchema goes 1 -> 2: the cached value's meaning changed while its key fields did not, so on a persistent backend shared with a released client each build would read the other's entry as its own. Precedent: RowIdSequenceKey (#8078).

Tests

20 in rust/lance/src/index.rs, one per site above. Every hunk in the PR was reverted individually to confirm a test fails without it.

Out of scope

unsupported_index_version falls back to i32::MAX for an unresolvable type_url, so a wholly unknown index type is reported as supported. Reversing it needs the system indices exempted first - neither fragment-reuse nor mem-wal details resolve to a scalar plugin - so it belongs in its own PR.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[wombatu-kun]

CI has been red on this PR only in unrelated infrastructure jobs, never in its own tests.

The run before the current one (33092361390) failed linux-build on uring::tests::test_read_range and uring::tests::test_read_multiple_ranges, both with no io_uring worker threads are available: ... Cannot allocate memory (os error 12). That is #8787, whose fix #8788 is green and waiting to merge. This PR touches nothing under lance-io. All 20 of its own tests passed in that same run, at positions 2505-2620 of 8823, before nextest fail-fast cancelled the remaining 1369.

Merged main again to retrigger. Wider context on the current flake wave is in #8789.

[wombatu-kun]

Both red jobs are known infrastructure flakes, not regressions from this branch. In this same run (33136954128, head 852e393c7) the linux-arm job ran both failing tests and both passed:

test index::vector::ivf::v2::tests::test_create_ivf_hnsw_pq_multivec ... ok
test uring::tests::test_large_file_read ... ok

windows-build in the same run executed the complete suite green (8557 tests run: 8557 passed, 12 skipped), including all 20 tests this PR adds.

job	test	cause
linux-build	lance-io uring::tests::test_large_file_read	no io_uring worker threads are available: thread 0: Cannot allocate memory (os error 12), runner environment, #8789 family 3
mac-build (stable)	lance index::vector::ivf::v2::tests::test_create_ivf_hnsw_pq_multivec	recall: 0.3 against the >= 0.5 bar, unseeded k-means init, #8764 (fix in flight as #8767)

Neither test's file is touched by this diff, which is confined to eight files under rust/lance/src/. #8764 already records this exact test at recall 0.3 on an unrelated PR.

I do not have write access here, so I cannot press "Re-run failed jobs". Could a maintainer re-run linux-build and mac-build (stable)?

[lance-gatekeeper]

✅ Gate recommendation: approve.

The complete/visible metadata split still holds across commit, cache, bookkeeping, planning, compaction, and validation after the latest base synchronization. Legacy no-bitmap coverage is reconstructed at its write version and mapped into current fragment IDs before planner exclusion and commit validation, preserving unsupported indices without permitting eager rewrites to strand them.