feat: support OPTIMIZE INDEX SQL

[majin1102]

Summary

• add ALTER TABLE ... OPTIMIZE INDEX SQL support
• maintain only the named index through Dataset.optimizeIndices() on the driver
• optionally pass through num_indices_to_merge; when omitted, leave merge policy to Lance Core
• enforce table writeability, reject system indexes, and preserve catalog-provided storage options
• derive output metrics from Lance Core index descriptions and statistics
• normalize and validate SQL options consistently across supported Spark versions
• add parser, end-to-end, read-only branch, error-path tests, and documentation

Size-tiered parameters are tracked separately in #744.

Closes #743

Testing

• ./mvnw spotless:apply
• Spark 3.4: OptimizeTest, LanceSqlExtensionsAstBuilderTest, and BranchDDLTest (63 tests)
• Spark 3.5: OptimizeTest, LanceSqlExtensionsAstBuilderTest, and BranchDDLTest (65 tests)
• Spark 4.2 / Scala 2.13 compile

[puchengy]

@majin1102 Hi, I didn't noticed we have a PR already, and I created my own version #755, do you plan to continue to work on this? Thanks

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-02T10:20:14.192408Z	c654ac5	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[majin1102]

@majin1102 Hi, I didn't noticed we have a PR already, and I created my own version #755, do you plan to continue to work on this? Thanks

Sorry for the late reply. This PR is now ready for review, and your feedback would be very welcome. There are also a few related issues we can look into together：#744, #750

[lance-gatekeeper]

✅ Gate recommendation: approve.

The two previously identified state-transition failures are closed. Empty MemWAL catch-up commits are handled as zero work, and this revision consumes Lance Core 12.0.0-beta.14, which includes the authoritative concurrent drop/maintenance conflict fix from lance#8984. The accompanying Namespace response adaptations preserve current connector behavior across the exercised Spark versions.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The targeted optimize-index path still consumes Core’s authoritative concurrent drop/maintenance guard, and this revision completes that Core 12 migration by declaring both position-delta transactions as whole-row rewrites. That matches their physical fragment replacement and Core conflict contract; Spark 3.5 delete, update, and merge coverage passes, and the Spark 3.4 source variant compiles.

Please mark this PR with the breaking-change label.

[majin1102]

@hamersaw @jackye1995 @zhangyue19921010

This PR is ready for review, please take a look when you have time