fix(substrait): decode deprecated timestamp literals as microseconds

[jonasdedden]

Fixes #8833. Also fixes #2514, which is the same bug reached through DuckDB and was closed without a fix.

Problem

A PyArrow filter on a timestamp column returned the wrong rows and raised nothing. pc.field("ts") > pa.scalar(datetime(2024, 1, 3, 2), pa.timestamp("us")) matched 0 of 100 rows where 49 was correct; the same filter written as SQL was fine. Timezone-aware columns failed the scan instead of answering wrongly.

Cause

PyArrow encodes the literal as Substrait's deprecated Literal.timestamp, defined by the spec as microseconds since the epoch, and leaves type_variation_reference at 0. DataFusion's consumer takes that field's unit from type_variation_reference and maps 0 to seconds, so the literal arrived a million times too large. On timestamp[us] the following cast overflows to null, which is why > and < both returned nothing. The deprecated Literal.timestamp_tz has no consumer branch at all.

Change

Before handing the expression to DataFusion, rewrite both deprecated literals into precision_timestamp / precision_timestamp_tz, which state the unit rather than implying it. Only the default reference changes meaning: references 1, 2 and 3 keep the milli/micro/nano units DataFusion gives them, and any other reference is left alone so DataFusion still reports it. Lance's own encode path uses the current DataFusion producer, which emits precision_timestamp, so it is unaffected.

remap_expr_references is renamed to normalize_expr because it now does more than remap field references. PyArrow could also be changed to emit the newer encoding; this handles the plans it produces today.

Tests

• rust/lance-datafusion/src/substrait.rs: parses the deprecated literal at each variation reference and asserts the resulting unit, plus the tz form that used to fail.
• python/python/tests/test_filter.py: >, < and == against timestamp[s|ms|us], naive and with a timezone, checked against PyArrow's own answer.

The DuckDB query from #2514 returns the matching row on this branch and an empty frame on pylance 10.0.0.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The Rust decode-boundary normalization matches Substrait’s microsecond contract for deprecated literals and fixes existing PyArrow inputs without duplicating semantics in Python. It preserves current precision literals and legacy non-default unit mappings; focused Rust and end-to-end PyArrow tests cover naive and timezone-aware comparisons.