feat: add ALTER TABLE ... OPTIMIZE INDEX for incremental index maintenance

[puchengy]

Summary

Implements #743 (sub-task of #742). Adds a SQL surface for incremental scalar-index maintenance. Today, indexing only the newly-appended (unindexed) fragments is reachable exclusively through the SDK Dataset.optimizeIndices. The SQL layer has no equivalent:

• ALTER TABLE ... CREATE INDEX <same name> is a full distributed rebuild over all fragments (atomically replaces the index).
• OPTIMIZE <table> compacts fragments only and does not train indexes.

This PR adds:

-- incrementally merge unindexed fragments into one index
ALTER TABLE lance.db.users OPTIMIZE INDEX idx_id;

-- optimize all indexes on the table
ALTER TABLE lance.db.users OPTIMIZE INDEX;

-- bound how many delta segments are merged in this run
ALTER TABLE lance.db.users OPTIMIZE INDEX idx_id WITH (num_indices_to_merge = 2);

It delegates to lance-core's Dataset.optimizeIndices, so it merges only unindexed fragments (existing coverage is not recomputed). Omitting the index name optimizes all user indexes. It runs on the driver (single node), consistent with the SDK path.

This keeps a clean separation of verbs: CREATE INDEX = define/rebuild, OPTIMIZE INDEX = incremental maintenance.

Options and validation

Option names are case-insensitive (normalized with Locale.ROOT, so they are stable under locales such as tr-TR). Supported WITH option:

Option	Type	Description
num_indices_to_merge	Integer >= 0 (default core-defined)	How many trailing delta index segments to fold together with the new data in this run; 0 appends the new data as a fresh delta segment instead of merging into existing ones.

The command validates its inputs at the Spark boundary rather than reporting work it did not do:

• A named index must exist and be a user index. A missing name, or a Lance system index (__lance_frag_reuse, __lance_mem_wal) that lance-core filters out before optimizing, is rejected instead of silently reported as optimized. Validation uses the same system-index filtering as SHOW INDEXES.
• Unknown options, duplicate options, wrong value types, and out-of-range num_indices_to_merge (int range) are rejected.
• retrain is not accepted here (see follow-ups).

Changes

• Grammar: new #optimizeIndex alternative in LanceSqlExtensions.g4 (reuses the existing OPTIMIZE / INDEX / WITH tokens; no new keywords).
• Plan/exec: OptimizeIndex logical plan + OptimizeIndexExec physical exec (modeled on DropIndexExec; validates WITH options and maps num_indices_to_merge to OptimizeOptions), wired into LanceDataSourceV2Strategy.
• AST builder: visitOptimizeIndex added across all Spark version modules (3.4 / 3.5 / 4.x).
• Shared helper: LanceSystemIndex centralizes the system-index name set + isSystemIndex, and locale-independent index-name normalization (normalizeName, Locale.ROOT). SHOW INDEXES now uses it too, and all three index DDLs (CREATE / DROP / OPTIMIZE INDEX) normalize names through it so the case-insensitive contract is consistent and locale-independent.
• Docs: create-index.md documents OPTIMIZE INDEX as the incremental path.

Testing

• LanceSqlExtensionsAstBuilderTest (3.4 + 3.5): index name, all-indexes (name omitted), and WITH (...) args parse cases.
• LanceSystemIndexTest: locale-independent normalization (incl. a tr-TR regression), isSystemIndex, and null handling.
• AddIndexTest (integration, SQL-driven): deferred zonemap populated via OPTIMIZE INDEX; missing-name rejected; system-index (__lance_frag_reuse) rejected; unknown option rejected; retrain rejected; case-insensitive option name honored.
• ShowIndexesTest: still green after the shared-helper refactor.
• All six Spark modules compile (compile on base / 3.4 / 3.5 / 4.2); spotless:check clean.
• Note: a few pre-existing AddIndexTest btree cases fail locally with a DataFusion external-sort OOM; these are environmental (constrained sandbox) and reproduce on main without this change.

Notes / follow-ups

• Incremental index maintenance currently runs single-node (inherited from optimizeIndices); making it distributed is a natural follow-up but out of scope here.
• retrain (rebuild an index's model from source data) is intentionally not exposed through this SQL command: in lance-core it applies only to v3 vector indexes and is a no-op for the scalar indexes this command targets, so exposing it via SQL would promise behavior it cannot deliver. It remains available via Dataset.optimizeIndices in the SDK. Exposing it for vector targets from SQL is a possible follow-up, ideally paired with lance-core returning an explicit error for unsupported targets (rather than a silent no-op) so the Spark layer does not have to re-encode the v3-vector contract. (retrain pass-through is listed in the Support OPTIMIZE INDEX SQL #743 scope.)
• Distributed execution of OPTIMIZE INDEX is tracked separately in Support distributed execution for OPTIMIZE INDEX #750; automatic size-tiered merge selection in Support size-tiered scalar index optimization #744.

🤖 Generated with Claude Code

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The SQL surface and driver-side delegation are a sound way to expose incremental index maintenance, but the new command can acknowledge work it did not perform and silently reinterpret its options. A viable revision should validate the named-index and WITH-option contract at the Spark boundary before calling lance-core, then report success only for a matched operation. This includes constraining and documenting retrain to the index types core supports.

[lance-gatekeeper]

The SQL option boundary silently changes valid requests: unquoted identifiers retain their spelling, but this map is queried only with lowercase literals, so WITH (RETRAIN = TRUE) is accepted and executes with retrain=false. Unknown keys are ignored, duplicates become last-write-wins, and Long.toInt wraps out-of-range values (4294967296 becomes 0), changing optimization behavior. Normalize unquoted option names and validate the allowed set, uniqueness, value types, and integer range before building the core options; also constrain and document retrain to the index families supported by core.

Reproducer

I ran a reflection test against this head that constructed:

OptimizeIndexExec(
  null, null, None,
  Seq(LanceNamedArgument("RETRAIN", true)))

and invoked buildOptions; asserting options.isRetrain failed because the observed value was false.

[lance-gatekeeper]

The general option validation is improved, but this contract root remains in two places.

• lance-core 11.0.0-beta.3 documents retrain as supported only for v3 vector indices. I ran a fully covered zonemap regression that recorded the dataset version, executed OPTIMIZE INDEX idx_retrain WITH (retrain = true), and asserted that the version advanced; SQL returned optimized, but the assertion failed with expected: <true> but was: <false>. The added deferred-zonemap test cannot distinguish retraining from ordinary incremental population. Reject or remove retrain for unsupported targets (including mixed all-index requests), or support it in core and add evidence from a real vector rebuild.
• arg.name.toLowerCase uses the process locale. A reflection regression set the default locale to tr-TR, passed RETRAIN, invoked buildOptions, and failed with Unsupported OPTIMIZE INDEX option(s): retraın. Use Locale.ROOT for normalization.

[lance-gatekeeper]

Option validation, retrain rejection, and option-name Locale.ROOT normalization are fixed. The same locale-sensitive normalization root remains in the newly added strategy branch: indexName.map(_.toLowerCase) uses the process locale. With default locale tr-TR, an existing idx_i targeted as IDX_I becomes ıdx_ı, and exact validation rejects it. Normalize this path with Locale.ROOT and keep a locale regression; aligning create/drop index normalization would keep the SQL index-name contract consistent.

Reproducer

I added a focused Spark 3.5 test that creates idx_i, switches the default locale to tr-TR, and executes ALTER TABLE ... OPTIMIZE INDEX IDX_I, then ran:

task_tmp_dir=$(mktemp -d /home/agent/tmp/gate755-index-locale.XXXXXX)
JAVA_TOOL_OPTIONS="-Djava.io.tmpdir=$task_tmp_dir" ./mvnw test -pl lance-spark-3.5_2.13 -Dtest="AddIndexTest#testOptimizeIndexNameCaseInsensitiveLocaleIndependent"

The test errored with Index ıdx_ı does not exist ... Existing indexes: idx_i from OptimizeIndexExec.

[lance-gatekeeper]

Fixed on 2702be8d: CREATE, DROP, and OPTIMIZE index names now share LanceSystemIndex.normalizeName, which uses Locale.ROOT. I reran the previously failing end-to-end Spark 3.5 scenario—create idx_i, switch to tr-TR, then optimize IDX_I—and it passed. The committed helper tests also pass on both Scala 2.12 and 2.13. Closing this finding.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The patch fixes ordinary missing-index handling and most WITH validation, but both prior contract roots remain: hidden system-index targets still report false success, and option handling still promises unsupported scalar retraining and depends on the JVM locale.

A viable revision should validate only user-optimizable index names, align retrain with the v3-vector-only contract in lance-core (including all-index requests), and normalize SQL option names with Locale.ROOT.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The missing/system-index and unsupported-option contracts are now fixed. One acceptance issue remains: case-insensitive index targeting still depends on the JVM default locale, so valid names can be rejected on affected installations.

Normalize index names with Locale.ROOT in the strategy path and retain the Turkish-locale regression; aligning create and drop index normalization will keep the SQL contract consistent.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The locale-dependent index-name path is now fixed, so all earlier acceptance findings are addressed. The SQL command preserves the intended incremental-maintenance contract: it validates user index names and options before delegating to lance-core, rejects unsupported retrain and system-index targets, and normalizes identifiers consistently across CREATE, DROP, and OPTIMIZE.