From 201a9e192a7ae1877a66dd2d3047dd4f58ec4291 Mon Sep 17 00:00:00 2001 From: Vivek Date: Fri, 28 Aug 2026 11:30:38 -0700 Subject: [PATCH] feat(format): specify carried-column storage and allow carrying a keyed column #8535 declared `IndexMetadata.covering_fields` but left the physical side unspecified; this specifies it -- carried values are extra columns in `auxiliary.idx`, discovered by exclusion against the quantizer's internal columns and bound to their source fields by a new `covering_field_ids` metadata key, with no `index_version` bump. It also permits an index to carry a column it is also keyed on, the only case where an id repeats in `fields`, so readers must take the carried set from `covering_fields` rather than subtract the keyed prefix. The "Current state" note becomes a rule about verifying each segment rather than a snapshot of which writers exist, so it stays accurate as implementations land instead of needing an edit to this spec each time one does. Adds `VectorQueryProto.covering_projection` (field 15) to reserve the tag, with the one initializer the new field forces on `query_to_proto`; no writer emits carried values yet, so the implementation follows separately. --- docs/src/format/index/index.md | 57 ++++++++++++++++++--------- docs/src/format/index/vector/index.md | 24 +++++++++++ protos/ann.proto | 23 +++++++++++ rust/lance/src/io/exec/ann_proto.rs | 3 ++ 4 files changed, 88 insertions(+), 19 deletions(-) diff --git a/docs/src/format/index/index.md b/docs/src/format/index/index.md index 01970133208..6121011e750 100644 --- a/docs/src/format/index/index.md +++ b/docs/src/format/index/index.md @@ -100,10 +100,11 @@ Index segments are created and updated through a transactional process: - `name`: The index name (must match existing segments if adding to an existing index) - `fields`: The columns the index depends on: the keyed column(s) it is searched on, followed by any merely-carried columns named in `covering_fields`. `fields[0]` is always a keyed column. - - `covering_fields`: The trailing subset of `fields` whose values the index carries but is not - keyed on, letting a query that only projects those columns be answered without a fragment take. - Empty for an index that carries no extra columns. Declaring a column here does not by itself - make it servable -- see [Serving carried columns](#serving-carried-columns). + - `covering_fields`: The trailing subset of `fields` whose values the index carries, letting a + query that only projects those columns be answered without a fragment take. Usually these are + columns the index is not keyed on, but a keyed column may also be carried. Empty for an index + that carries no extra columns. Declaring a column here does not by itself make it servable -- + see [Serving carried columns](#serving-carried-columns). - `fragment_bitmap`: The set of fragment IDs covered by this segment - `index_details`: Index-specific configuration and parameters - `version`: The format version of this index type @@ -141,18 +142,36 @@ fragments that would have been covered by that segment. carries. It does not establish that the segment's storage holds their values. **The segment's storage schema is authoritative.** Before answering a query from a -carried column, an engine must confirm that column is present in the storage it opened, -and fall back to a take against the base table when it is not. A segment whose -declaration names a column its storage does not hold is a legal state, not corruption: -a maintenance operation that cannot carry the payload through a rebuild is permitted to -withdraw it and leave the declaration standing. - -!!! note "Current state" - - No index builder writes carried values yet, so today every declaration is ahead of - its storage. Engines that read `covering_fields` must therefore treat it purely as a - declaration and serve every column from the base table until they have verified the - storage themselves. This is transitional; the rule above is not. +carried column, an engine must confirm that column is present and bound to the declared +logical field in the storage it opened, and fall back to a take against the base table +when it cannot. A segment whose declaration names a column its storage does not hold is +a legal state, not corruption: a maintenance operation that cannot carry the payload +through a rebuild is permitted to withdraw it and leave the declaration standing. + +!!! note "Capability varies by segment" + + Whether a segment's storage holds a declared column depends on the index type, on the + writer that produced the segment, and on what later maintenance did to it, so one + logical index may hold values for some of its segments and not others. An engine + therefore verifies each selected segment rather than inferring capability from the + index type, the writer version, or the declaration alone, and serves from the base + table every column it cannot verify. + +!!! note "A keyed column may also be carried" + + `covering_fields` usually names columns the index is *not* keyed on, but an index is + permitted to carry a column it is also keyed on -- for instance a vector index that + keeps full-precision vectors so a refine pass can re-rank without a base-table take. + The id then appears twice in `fields`, once as `fields[0]` and again as the trailing + carried entry, and once in `covering_fields`. This is the only case in which an id + repeats in `fields`. + + A reader must therefore take the carried set from `covering_fields` directly, and + never derive it by subtracting the keyed prefix from `fields`: that set difference + silently drops a column that is both. The trailing-subset rule is stated over + `covering_fields` and is unaffected: `fields[0]` remains the column the index is + searched on, and an engine serves the repeated id from storage like any other + carried column. ## Loading an index @@ -175,9 +194,9 @@ The `IndexMetadata` message contains important information about the index segme - `fields`: the columns the index depends on: the keyed column(s) the index is searched on, followed by any columns it merely carries, as named in `covering_fields`. `fields[0]` is always a keyed column. - `covering_fields`: the trailing subset of `fields` whose values the index carries alongside its own - data but is not keyed on. Empty for an index that carries no extra columns. This declaration is - not authoritative for what the segment can serve -- see - [Serving carried columns](#serving-carried-columns). + data -- usually columns it is not keyed on, though a keyed column may also be carried. Empty for an + index that carries no extra columns. This declaration is not authoritative for what the segment can + serve -- see [Serving carried columns](#serving-carried-columns). - `fragment_bitmap`: the set of fragment IDs covered by this index segment. - `index_details`: a protobuf `Any` message that contains index-specific details, such as index type, parameters, and storage format. This allows different index types to store their own metadata. diff --git a/docs/src/format/index/vector/index.md b/docs/src/format/index/vector/index.md index 91d73a23a16..709b9a58cc7 100644 --- a/docs/src/format/index/vector/index.md +++ b/docs/src/format/index/vector/index.md @@ -161,6 +161,19 @@ the Arrow schema of the Lance file varies depending on the quantization method u !!! note All partitions are stored in the same file, and partitions must be written in order. +Every quantization format below lists only its internal columns. When a V3 IVF +writer materializes carried values, it appends one trailing column per carried +field after them, named and typed exactly as in the dataset schema. This physical +payload may be a subset of the manifest's `covering_fields` declaration (see +[Index Metadata](../index.md)). A reader returns only columns whose physical +schema and source field ids it verifies across every selected segment; all other +projected columns come from a base-table take. + +A reader discovers carried columns by exclusion, not by position: any column in the +auxiliary file's schema that is not one of the quantizer's internal columns is a +carried column. Writers append them in trailing order, but a reader must not depend +on that ordering to identify them. + ##### FLAT No quantization applied - stores original vectors in their full precision: @@ -229,6 +242,17 @@ Contains RabitQ-specific metadata in JSON format (only present for RQ quantizati This includes the rotation matrix position, number of bits, and packing information. See the RQ metadata specification in the "storage_metadata" section below. +##### "covering_field_ids" + +The *source dataset* field ids of the storage file's physical carried columns, +comma separated in physical schema order (only present when the storage carries +values). Arrow fields carry no Lance field id, so names and types alone cannot +prove which logical column a payload came from. Readers use these ids to bind +physical values to the segment's `covering_fields` declaration, and treat missing, +malformed, ambiguous, or mismatched metadata as no servable carried capability. +Distributed merges use the same identity to reject shards whose columns match by +name and type but come from different fields. + ##### "storage_metadata" Contains quantizer-specific metadata as a list of JSON strings. diff --git a/protos/ann.proto b/protos/ann.proto index f5de5e25e7b..fd91c3c6d27 100644 --- a/protos/ann.proto +++ b/protos/ann.proto @@ -23,6 +23,23 @@ enum VectorApproxMode { Accurate = 2; } +// The covering ("included") index columns a query needs materialized, by name. +// +// Exists as a message rather than a bare `repeated string` because proto3 has no +// presence tracking for repeated fields, and the empty list is a distinct, meaningful +// state here: +// +// * absent: no narrowing computed; materialize every covering column declared. +// * present and empty: materialize nothing, though the index does declare covering. +// * present and non-empty: materialize exactly these. +// +// Collapsing "present, empty" into "absent" costs no correctness (a covering column is +// semantically transparent -- its values otherwise arrive from a base-table read) but +// silently restores full materialization on every distributed plan. +message CoveringProjection { + repeated string columns = 1; +} + // Serialized vector query parameters. message VectorQueryProto { // Query vector as Arrow IPC bytes (supports Float16, Float32, Float64, UInt8, etc.) @@ -43,6 +60,12 @@ message VectorQueryProto { // Query-time approximation mode. Currently only affects RQ-quantized vector // indexes, such as IVF_RQ. Other index types ignore this setting. VectorApproxMode approx_mode = 14; + // Which covering columns the index must materialize for this query. Absent means + // "not computed" -- see CoveringProjection. Carried across the wire so a remote + // executor declares the same search output schema the planner did; without it the + // executor's node is wider than the plan it came from, and the surrounding nodes + // were built against the planner's narrower schema. + CoveringProjection covering_projection = 15; } // Serializable form of ANNIvfSubIndexExec — the IVF sub-index search node. diff --git a/rust/lance/src/io/exec/ann_proto.rs b/rust/lance/src/io/exec/ann_proto.rs index c57ad4ca1b7..7ecb14211e6 100644 --- a/rust/lance/src/io/exec/ann_proto.rs +++ b/rust/lance/src/io/exec/ann_proto.rs @@ -118,6 +118,9 @@ pub fn query_to_proto(query: &Query) -> Result { dist_q_c: Some(query.dist_q_c), query_parallelism: Some(query.query_parallelism), approx_mode: approx_mode_to_proto(query.approx_mode) as i32, + // No planner narrows the covering projection yet, so this is always absent: + // "materialize every covering column declared". See `CoveringProjection`. + covering_projection: None, }) }