feat(format): specify carried-column storage and allow carrying a keyed column

docs/src/format/index/index.md

@@ -100,10 +100,11 @@ Index segments are created and updated through a transactional process:
    - `name`: The index name (must match existing segments if adding to an existing index)
    - `fields`: The columns the index depends on: the keyed column(s) it is searched on, followed
      by any merely-carried columns named in `covering_fields`. `fields[0]` is always a keyed column.
-   - `covering_fields`: The trailing subset of `fields` whose values the index carries but is not
-     keyed on, letting a query that only projects those columns be answered without a fragment take.
-     Empty for an index that carries no extra columns. Declaring a column here does not by itself
-     make it servable -- see [Serving carried columns](#serving-carried-columns).
+   - `covering_fields`: The trailing subset of `fields` whose values the index carries, letting a
+     query that only projects those columns be answered without a fragment take. Usually these are
+     columns the index is not keyed on, but a keyed column may also be carried. Empty for an index
+     that carries no extra columns. Declaring a column here does not by itself make it servable --
+     see [Serving carried columns](#serving-carried-columns).
    - `fragment_bitmap`: The set of fragment IDs covered by this segment
    - `index_details`: Index-specific configuration and parameters
    - `version`: The format version of this index type
@@ -141,18 +142,36 @@ fragments that would have been covered by that segment.
 carries. It does not establish that the segment's storage holds their values.
 
 **The segment's storage schema is authoritative.** Before answering a query from a
-carried column, an engine must confirm that column is present in the storage it opened,
-and fall back to a take against the base table when it is not. A segment whose
-declaration names a column its storage does not hold is a legal state, not corruption:
-a maintenance operation that cannot carry the payload through a rebuild is permitted to
-withdraw it and leave the declaration standing.
-
-!!! note "Current state"
-
-    No index builder writes carried values yet, so today every declaration is ahead of
-    its storage. Engines that read `covering_fields` must therefore treat it purely as a
-    declaration and serve every column from the base table until they have verified the
-    storage themselves. This is transitional; the rule above is not.
+carried column, an engine must confirm that column is present and bound to the declared
+logical field in the storage it opened, and fall back to a take against the base table
+when it cannot. A segment whose declaration names a column its storage does not hold is
+a legal state, not corruption: a maintenance operation that cannot carry the payload
+through a rebuild is permitted to withdraw it and leave the declaration standing.
+
+!!! note "Capability varies by segment"
+
+    Whether a segment's storage holds a declared column depends on the index type, on the
+    writer that produced the segment, and on what later maintenance did to it, so one
+    logical index may hold values for some of its segments and not others. An engine
+    therefore verifies each selected segment rather than inferring capability from the
+    index type, the writer version, or the declaration alone, and serves from the base
+    table every column it cannot verify.
+
+!!! note "A keyed column may also be carried"
+
+    `covering_fields` usually names columns the index is *not* keyed on, but an index is
+    permitted to carry a column it is also keyed on -- for instance a vector index that
+    keeps full-precision vectors so a refine pass can re-rank without a base-table take.
+    The id then appears twice in `fields`, once as `fields[0]` and again as the trailing
+    carried entry, and once in `covering_fields`. This is the only case in which an id
+    repeats in `fields`.
+
+    A reader must therefore take the carried set from `covering_fields` directly, and
+    never derive it by subtracting the keyed prefix from `fields`: that set difference
+    silently drops a column that is both. The trailing-subset rule is stated over
+    `covering_fields` and is unaffected: `fields[0]` remains the column the index is
+    searched on, and an engine serves the repeated id from storage like any other
+    carried column.
 
 ## Loading an index
 
@@ -175,9 +194,9 @@ The `IndexMetadata` message contains important information about the index segme
 - `fields`: the columns the index depends on: the keyed column(s) the index is searched on, followed
   by any columns it merely carries, as named in `covering_fields`. `fields[0]` is always a keyed column.
 - `covering_fields`: the trailing subset of `fields` whose values the index carries alongside its own
-  data but is not keyed on. Empty for an index that carries no extra columns. This declaration is
-  not authoritative for what the segment can serve -- see
-  [Serving carried columns](#serving-carried-columns).
+  data -- usually columns it is not keyed on, though a keyed column may also be carried. Empty for an
+  index that carries no extra columns. This declaration is not authoritative for what the segment can
+  serve -- see [Serving carried columns](#serving-carried-columns).
 - `fragment_bitmap`: the set of fragment IDs covered by this index segment.
 - `index_details`: a protobuf `Any` message that contains index-specific details, such as index type,
   parameters, and storage format. This allows different index types to store their own metadata.

docs/src/format/index/vector/index.md

@@ -161,6 +161,19 @@ the Arrow schema of the Lance file varies depending on the quantization method u
 !!! note
     All partitions are stored in the same file, and partitions must be written in order.
 
+Every quantization format below lists only its internal columns. When a V3 IVF
+writer materializes carried values, it appends one trailing column per carried
+field after them, named and typed exactly as in the dataset schema. This physical
+payload may be a subset of the manifest's `covering_fields` declaration (see
+[Index Metadata](../index.md)). A reader returns only columns whose physical
+schema and source field ids it verifies across every selected segment; all other
+projected columns come from a base-table take.
+
+A reader discovers carried columns by exclusion, not by position: any column in the
+auxiliary file's schema that is not one of the quantizer's internal columns is a
+carried column. Writers append them in trailing order, but a reader must not depend
+on that ordering to identify them.
+
 ##### FLAT
 
 No quantization applied - stores original vectors in their full precision:
@@ -229,6 +242,17 @@ Contains RabitQ-specific metadata in JSON format (only present for RQ quantizati
 This includes the rotation matrix position, number of bits, and packing information.
 See the RQ metadata specification in the "storage_metadata" section below.
 
+##### "covering_field_ids"
+
+The *source dataset* field ids of the storage file's physical carried columns,
+comma separated in physical schema order (only present when the storage carries
+values). Arrow fields carry no Lance field id, so names and types alone cannot
+prove which logical column a payload came from. Readers use these ids to bind
+physical values to the segment's `covering_fields` declaration, and treat missing,
+malformed, ambiguous, or mismatched metadata as no servable carried capability.
+Distributed merges use the same identity to reject shards whose columns match by
+name and type but come from different fields.
+
 ##### "storage_metadata"
 
 Contains quantizer-specific metadata as a list of JSON strings.

protos/ann.proto

@@ -23,6 +23,23 @@ enum VectorApproxMode {
   Accurate = 2;
 }
 
+// The covering ("included") index columns a query needs materialized, by name.
+//
+// Exists as a message rather than a bare `repeated string` because proto3 has no
+// presence tracking for repeated fields, and the empty list is a distinct, meaningful
+// state here:
+//
+// * absent: no narrowing computed; materialize every covering column declared.
+// * present and empty: materialize nothing, though the index does declare covering.
+// * present and non-empty: materialize exactly these.
+//
+// Collapsing "present, empty" into "absent" costs no correctness (a covering column is
+// semantically transparent -- its values otherwise arrive from a base-table read) but
+// silently restores full materialization on every distributed plan.
+message CoveringProjection {
+  repeated string columns = 1;
+}
+
 // Serialized vector query parameters.
 message VectorQueryProto {
   // Query vector as Arrow IPC bytes (supports Float16, Float32, Float64, UInt8, etc.)
@@ -43,6 +60,12 @@ message VectorQueryProto {
   // Query-time approximation mode. Currently only affects RQ-quantized vector
   // indexes, such as IVF_RQ. Other index types ignore this setting.
   VectorApproxMode approx_mode = 14;
+  // Which covering columns the index must materialize for this query. Absent means
+  // "not computed" -- see CoveringProjection. Carried across the wire so a remote
+  // executor declares the same search output schema the planner did; without it the
+  // executor's node is wider than the plan it came from, and the surrounding nodes
+  // were built against the planner's narrower schema.
+  CoveringProjection covering_projection = 15;
 }
 
 // Serializable form of ANNIvfSubIndexExec — the IVF sub-index search node.

rust/lance/src/io/exec/ann_proto.rs

@@ -118,6 +118,9 @@ pub fn query_to_proto(query: &Query) -> Result<pb::VectorQueryProto> {
         dist_q_c: Some(query.dist_q_c),
         query_parallelism: Some(query.query_parallelism),
         approx_mode: approx_mode_to_proto(query.approx_mode) as i32,
+        // No planner narrows the covering projection yet, so this is always absent:
+        // "materialize every covering column declared". See `CoveringProjection`.
+        covering_projection: None,
     })
 }