feat(sql): add REPLACE ... WHERE for atomic predicate-scoped overwrite

[puchengy]

What

Adds a REPLACE <table> WHERE <predicate> AS <query> SQL command that atomically replaces the rows of a Lance table matching <predicate> with the result of <query>, in a single table version.

REPLACE lance.db.events
  WHERE dt = '2026-08-01'
  AS SELECT id, dt, value FROM staging_events WHERE dt = '2026-08-01';

Why

This is the predicate-scoped analogue of Iceberg's INSERT OVERWRITE ... PARTITION(...). Lance has no partition spec, so today the only ways to replace part of a table are:

• DELETE then INSERT — two separate commits, so a reader can observe the region deleted-but-not-reinserted, and a crash between them leaves it permanently missing.
• whole-table INSERT OVERWRITE — atomic but replaces the entire table.
• MERGE INTO — atomic, but matching on non-unique columns (e.g. a date) is a per-partition cartesian join.

REPLACE ... WHERE fills the gap: a single atomic delete + append scoped by an arbitrary predicate.

How

The delete of matching rows and the append of new rows are committed as one Lance Operation.Update{removedFragmentIds, updatedFragments, newFragments} — so readers never see a half-applied state, and a failure cannot leave the region half-written.

• Fragments fully covered by the predicate are dropped (removedFragmentIds).
• Fragments that partially match keep their non-matching rows via a deletion vector (updatedFragments), so a fragment holding multiple dt values is handled correctly.
• A predicate matching no existing rows degrades to a plain append.

Implementation reuses the existing distributed write pipeline. The predicate rides through as an internal write option and LanceBatchWrite.commit() branches to assemble the atomic Update (the same commit primitive already used by the row-level position-delta path). Grammar captures the predicate and query as raw text and re-parses the query with Spark's own parser via the delegate.

Files

• Grammar: LanceSqlExtensions.g4 (new REPLACE ... WHERE ... AS rule; adds a WS skip rule + catch-all token so the raw predicate/query regions tokenize cleanly).
• Parser: visitReplaceWhere added to the AST builder in all 5 per-version modules (3.4 / 3.5 / 4.0 / 4.1 / 4.2).
• Plan/strategy/exec: ReplaceWhere logical plan, LanceDataSourceV2Strategy case, ReplaceWhereExec.
• Commit: LanceBatchWrite.buildReplaceOperation + a replaceWhere option on LanceSparkWriteOptions (excluded from toWriteParams, never forwarded to native).
• Docs: docs/src/operations/dml/replace.md.

Testing

New ReplaceWhereTest (base test in lance-spark-base, run under 3.5): single-partition replace, other-partitions-untouched, partially-matching fragment (deletion vector), replace of a non-existing partition (append), and single-atomic-commit (exactly one version bump). Ran spotless:check + checkstyle:check (clean) and the full base + 3.5 suite locally; all pass except pre-existing environmental OOMs in AddIndexTest (external-sort memory limit in the sandbox), which are unrelated to this change.

🤖 Generated with Claude Code

[puchengy]

Pushed 498fdef addressing both gate-keeper blockers:

1. Top-level AS split — the grammar now captures the whole post-WHERE body as one raw region and splits it at the first top-level AS in ParserUtils.splitReplaceBody (paren/quote/comment-aware), so predicates containing AS (e.g. CAST(dt AS STRING)) work. Added a CAST-in-predicate regression test (testReplacePredicateWithCast).
2. Bounded deletion memory — deletions are now accumulated as a compressed RoaringBitmap per fragment and materialized to a row-index list one fragment at a time, so driver memory no longer grows with total matched row count. Mirrors the existing SparkPositionDeltaWrite pattern.

Local: spotless + checkstyle clean; ReplaceWhereTest (6 tests incl. CAST) passes on 3.5.

[puchengy]

Pushed a79e7b6 for the second-round findings:

1. Nested block comments — splitReplaceBody block-comment skipping is now depth-aware, so an AS inside an outer comment (/* outer /* inner */ AS ... */) is no longer mistaken for the separator. Added testReplacePredicateWithNestedBlockComment.
2. Full-match fragment fast path — a fragment whose entire live row count matches is now dropped straight to removedFragmentIds without materializing any per-row list (the common whole-partition-per-fragment case, memory independent of fragment size). Only genuinely partial fragments materialize a list, and that is bounded by a single fragment's row count — the inherent limit of the native deleteRows(List<Integer>) API. A fully general partial-match path would need a bitmap-accepting deleteRows in lance-core, which is outside this connector PR.

Local: spotless + checkstyle clean; ReplaceWhereTest (7 tests) passes on 3.5.

[puchengy]

Pushed 4ce419b.

Line comments (fixed): splitReplaceBody now ends a line comment at either \r or \n, matching Spark. Added testReplacePredicateWithCarriageReturnLineComment. Also switched the full-match check to getLongCardinality().

Partial-fragment materialization (won't fix in this PR): I don't think this is a real concern, and I'd like to resolve it as accepted rather than block on it. The reasoning:

• A boxed Integer + its ArrayList slot is ~20 bytes on a 64-bit JVM (compressed oops), so 1M matched rows ≈ 20 MB and 15M ≈ 300 MB. The reproducer only OOMs because it pins -Xmx256m; this list is built on the Spark driver, whose heap is normally multiple GB, so 300 MB is not a realistic OOM.
• The common partition-overwrite case (a whole partition in its own fragment) hits the full-match fast path and materializes nothing — it's dropped by fragment id.
• The residual path only triggers for a fragment that is simultaneously (a) partially matched and (b) very large. Fully eliminating boxing there is not possible in the connector: the native Fragment.deleteRows(List<Integer>) binding accepts only a boxed list. A bounded fix requires a bitmap/primitive-accepting deleteRows in lance-core, which is a separate repo/release.

Proposal: land this connector PR with the full-match fast path (covers the intended partition-aligned workload with bounded memory), and track a bitmap-accepting deleteRows as a lance-core follow-up for the large-partial-fragment tail case. Happy to file that issue.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[puchengy]

The one red check (Integration Test Spark 3.4 / Scala 2.12) is an infra flake, not a code issue — Maven Central returned HTTP 403 Forbidden for maven-install-plugin:2.5.2 during make bundle, before any test ran. Tracked in #760. It needs a maintainer to re-run the failed job (fork contributors can't); the change itself is unaffected.

[puchengy]

Tracking issue for this feature: #762 (support partition-scoped INSERT OVERWRITE semantics via SQL). It captures the motivation (Hive/Iceberg migration relying on partition-scoped INSERT OVERWRITE), why MERGE INTO can't substitute, and the proposed REPLACE ... WHERE surface built on the format's atomic Update transaction.

[puchengy]

Pushed 497011c implementing the metadata-only fragment removal follow-up (previously noted on #762).

For a REPLACE ... WHERE whose predicate is a conjunction of column = literal on zonemap-indexed columns, fragments the zonemap proves are fully covered (every zone min==max==value, no nulls) are dropped by id without scanning their rows; unproven fragments are scanned exactly as before, restricted to their ids. When all matching fragments are covered, the delete-planning scan is skipped entirely — replacing a partition that occupies its own fragments is now metadata-only (O(#fragments), not O(rows)).

• ReplaceWhereExec parses the predicate with Spark's parser and only emits the equality terms (as an internal option) when it's a pure conjunction of equalities on string/integral columns.
• ReplaceCoverage proves coverage from getZonemapStats; multiple single-column zonemaps combine by intersection for multi-column predicates (e.g. dt + hr).
• Fail-safe: any missing zonemap, unproven zone, non-equality predicate, or value/format mismatch defers to the exact scan, so the optimization never changes which rows are replaced.

New tests cover the covered-partition drop, multi-column zonemap, and range-predicate fallback (each asserting results identical to the scan path). 12 ReplaceWhereTest cases pass; spotless + checkstyle clean; 2.12 and 2.13 main sources compile.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new zonemap optimization regresses case-insensitive equality predicates before the exact fallback. Restore fail-safe behavior by resolving target columns canonically or treating an ineligible metadata lookup as a scan fallback.

The product contract from the prior review also remains open. Maintainers still need to choose general Lance predicate replacement versus Spark overwrite-by-filter or partition overwrite, and define inserted-row constraints plus Spark-versus-Lance predicate semantics. The single-version Update direction and the previously accepted large-partial-fragment driver-memory risk are otherwise unchanged.

[lance-gatekeeper]

This metadata lookup is not fail-safe for case-insensitive column references. ReplaceWhereExec serializes the raw UnresolvedAttribute spelling, but Lance's filter planner resolves ordinary names against the schema case-insensitively while getZonemapStats requires the exact schema name. Consequently, WHERE DT = ... on a dt column now aborts here before the exact scan, even when no zonemap exists. Canonicalize the equality column against the target schema before probing, or treat lookup failure as ineligible for the optimization and fall back to the exact scan; retain a mixed-case regression.

Reproducer

I added and ran this test against 497011c:

@Test
public void testReplacePredicateColumnIsCaseInsensitive() {
  TableOperator op = new TableOperator(spark, catalogName);
  op.create();
  op.insert(Arrays.asList(Row.of(1, "2026-08-01", 100)));

  op.replace(
      "DT = '2026-08-01'",
      "SELECT 2 AS id, '2026-08-01' AS dt, 200 AS value");

  op.check(Arrays.asList(Row.of(2, "2026-08-01", 200)));
}

./mvnw -pl lance-spark-3.5_2.12 -Dtest=ReplaceWhereTest#testReplacePredicateColumnIsCaseInsensitive -Djava.io.tmpdir=/home/repo/.gate-tmp.PKyukZ test

Expected: the case-insensitive predicate replaces the row. Observed: IllegalArgumentException: Column 'DT' not found in dataset schema from this call; the submitted 12-test suite passes because every equality uses exact-case column names.