fix(linalg): reject a null Int8 query instead of panicking

[LuciferYang]

What this changes

l2_distance_arrow_batch, dot_distance_arrow_batch and cosine_distance_arrow_batch take the query as a &dyn Array and widen an Int8 one element at a time. All three did it the same way:

DataType::Int8 => do_l2_distance_arrow_batch::<Float32Type>(
    &from
        .as_primitive::<Int8Type>()
        .into_iter()
        .map(|x| x.unwrap() as f32)
        .collect(),
    &to.convert_to_floating_point()?,
),

A null element in the query reaches that unwrap and panics, in every profile. The three functions all return Result and all document that the null buffer of to is propagated to the output, which says nothing about nulls in from, so a caller has no reason to expect a panic from the other side.

The three sites now share int8_query_to_f32, which rejects the input with InvalidArgumentError and names the count, then widens through values() so no Option remains. Returning an error rather than panicking is what the unsupported-type arm of the same match already does, though l2's is a ComputeError where dot's and cosine's are InvalidArgumentError. The arm is reachable from a real caller: lance-index's flat search passes a user query straight into DistanceType::arrow_batch_func().

Three smaller things travel with it.

dot_distance_arrow_batch carried a debug_assert_eq! on the dimension in its public entry point, ahead of the Int8 arm. Once that arm returns an error instead of panicking, the copy preempts it in a debug build: a query that is both null-bearing and the wrong length would panic on dot and return the error on its two siblings. The copy is gone; do_dot_distance_arrow_batch still carries the same assert, which is where l2 and cosine have theirs.

All three functions gain an # Errors section, which none of them had. It names the new null rejection and says the list is not exhaustive, since the unsupported-type and downcast arms return errors of their own.

All three already had a # Panics section, and only cosine's changes: it now says the length mismatch is caught with debug assertions on, and not reliably without them, because cosine has no always-on layout assert of its own. l2 and dot keep theirs as written, since assert_batch_layout is a plain assert! on every batch path they reach.

The float arms are untouched. Float16, Float32 and Float64 hand the array to do_*_arrow_batch directly, which reads it through as_slice() and ignores the validity buffer, so a null there is read as whatever the slot holds rather than panicking. Changing that is a separate decision about what a null query element should mean.

Test plan

test_arrow_batch_rejects_null_int8_query goes through DistanceType::arrow_batch_func() for L2, Cosine and Dot, so it covers all three sites through the public dispatch rather than each function directly. Three parts:

• a query of [Some(1), None] returns InvalidArgumentError whose message contains both Int8 query vector \from`andfound 1 in 2 values`
• the same query without nulls returns two distances, so the guard cannot pass by rejecting every Int8 query
• a query sliced out of [None, Some(3), Some(4), None] at offset 1 goes through, because the slice window holds no nulls even though the full buffer does. Both the null count and the widening have to describe the same window or this either rejects a clean query or widens the wrong two elements. L2's result is asserted literally as [8.0, 0.0], which is what pins the widening; Cosine and Dot are compared against a call on an unsliced [3, 4], which pins the window but would agree with itself if the widening were wrong.

test_arrow_batch_null_and_length_mismatch_agree sends an input that is wrong in both ways at once, a three-element null-bearing query against a dimension-2 target, and asserts all three metrics reach the null error. Restoring dot's entry-point debug_assert_eq! makes that test fail on dot with left: 3, right: 2.

Measured: restoring the three unwrap call sites makes test_arrow_batch_rejects_null_int8_query fail with a panic at the unwrap in l2.rs rather than an error.

• cargo test -p lance-linalg: 396 passed, 1 ignored
• cargo fmt --all -- --check and cargo clippy -p lance-linalg --all-targets -- -D warnings: clean

[LuciferYang]

My previous revision made the # Panics line worse, and this reverts that half.

I had qualified it with "With debug assertions on", on the theory that the debug_assert_eq!(from.len(), dimension) was the only length check. It is only the first one. With debug assertions off the batch kernels assert unconditionally underneath it, so a length mismatch still panics. Measured with RUSTFLAGS="-C debug-assertions=off" on all three metrics: l2 and dot give distance vector length must match dimension: vector=3, dimension=2 from assert_batch_layout, and cosine gives distance inputs must have equal lengths: left=3, right=2 from assert_equal_lengths reached through cosine_fast and dot. The line is unconditional again, keeping only the clause that one of the documented errors can be returned first.

Worth noting how nearly I convinced myself otherwise: my first attempt at that measurement used an Int8 target array with a Float32 query, which fails the downcast before it reaches any length check, so it looked like release did not panic. The element types have to match for the test to reach the code under test.

The comment in dot.rs had the same profile dependency and is now structural instead: the duplicate assert put the dimension check ahead of the Int8 arm null check, which neither l2 nor cosine does.

[LuciferYang]

A review pass found that the # Panics line I unqualified in all three files is right for two of them and wrong for the third, so it is now per-file.

l2_batch and dot_batch open with assert_batch_layout, whose assert_eq!(vector_len, dimension) is not profile-gated, so an unqualified "panics on a length mismatch" is true for l2 and dot in every build. Cosine calls that helper nowhere: its only dimension check is debug_assert_eq!(from.len(), dimension), and f32::cosine_batch does let _ = batch.chunks_exact(dimension), which validates nothing about x.

Measured on the same crate in both profiles. With a dimension-8 target and a length-3 f32 query, debug panics and release returns [-1.27e20, -1.51e20] from all-small-positive inputs, so it read past the query buffer rather than merely computing on a short vector. Same for an Int8 query of length 16 against dimension 8, which is the arm this PR owns: no nulls, wrong length, silent garbage in release. l2 and dot panicked on every one of those inputs in release. The dimension-2 cases panic in release too, but incidentally, through dots own assert in the scalar tail, which the dimension-8 arms never reach.

The line for cosine now says the check is debug-only and names why. That describes this PR base; #8875 adds assert_batch_layout to both cosine batch entries, and once it lands the unqualified wording becomes correct there too.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The author’s correction matches the current paths: cosine’s dimension check is debug-only, whereas L2 and Dot use the always-on layout assertion. The revised docs now describe that pre-existing mismatch hazard accurately; it is unchanged by this null-query fix and is addressed separately by #8875, so no additional change is needed here. The shared Int8 null guard and focused regression still behave as intended.

[LuciferYang]

Fifth instance of the same class, and the most interesting one. My cosine qualifier said a mismatch "can read past the shorter vector" in release. Literally hedged, but the reasoning behind it was wrong: cosine having no always-on layout assert does not mean nothing catches a mismatch.

Measured on aarch64, both profiles. f32::cosine_batch branches on the dimension: 8 and 16 go to cosine_once_8 / cosine_once_16, which do the unchecked load_unaligned(x.as_ptr()), and every other dimension falls to cosine_fast → cosine_scalar → dot, whose assert_equal_lengths is a plain assert_eq!. So with debug assertions off, dimension 2 with a length-3 query panics with distance inputs must have equal lengths, while dimension 8 with a length-4 query returns [0.617, 0.441] and dimension 16 with a length-8 query returns [-0.260, -0.297]. Both of this PR's own tests use dimension 2, which is the case that does panic.

The line now says the mismatch is not reliably caught without debug assertions, which holds at every dimension, rather than asserting a read-past that only happens at two of them. Naming the dimensions in the doc would be a case table, and #8875 removes the distinction anyway by adding assert_batch_layout to both cosine batch entries.

Not claimed, since it was not run: x86_64. There the sub-AVX2 CosineBatchIter and the AVX-512 cosine_batch_avx512 arm both take dimension and read that many key elements unchecked, so the read-past window there is probably not limited to 8 and 16.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The latest correction matches the dimension-dependent paths: without debug assertions, specialized cosine branches can bypass a length check while fallback branches may still panic through dot. The revised “not reliably caught” wording now describes that pre-existing behavior accurately. The shared Int8 null guard and regression remain unchanged and valid; #8875 independently adds the always-on layout assertion.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The latest revision only tightens internal explanations: the helper is now scoped to the three batch entry points that actually accept Int8, and the Dot comments name the duplicate debug_assert_eq! that previously ran before the null guard. These statements match the dispatch and call order. Production behavior, the shared null guard, and its regression remain unchanged; the separate cosine layout fix is still handled independently in #8875.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The latest revision correctly avoids promising validation precedence while retaining accurate error and panic contracts. The shared Int8 conversion guard still turns null query coordinates into a descriptive error for Cosine, Dot, and L2 without changing supported-query results. The pre-existing cosine release layout gap remains independently handled by #8875.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The latest cleanup removes edit-history commentary and restores unchanged documentation wrapping without altering executable behavior or public contracts. The shared Int8 conversion guard still rejects null query coordinates with descriptive errors across Cosine, Dot, and L2 while preserving valid and sliced inputs. The separate cosine layout gap remains independently handled by #8875.