feat(format): specify carried-column storage and allow carrying a keyed column - #8856
Open
vivek-bharathan wants to merge 1 commit into
Open
feat(format): specify carried-column storage and allow carrying a keyed column#8856vivek-bharathan wants to merge 1 commit into
vivek-bharathan wants to merge 1 commit into
Conversation
github-actions
Bot
added
A-format
Contributor
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
…ed column lance-format#8535 declared `IndexMetadata.covering_fields` but left the physical side unspecified; this specifies it -- carried values are extra columns in `auxiliary.idx`, discovered by exclusion against the quantizer's internal columns and bound to their source fields by a new `covering_field_ids` metadata key, with no `index_version` bump. It also permits an index to carry a column it is also keyed on, the only case where an id repeats in `fields`, so readers must take the carried set from `covering_fields` rather than subtract the keyed prefix. The "Current state" note becomes a rule about verifying each segment rather than a snapshot of which writers exist, so it stays accurate as implementations land instead of needing an edit to this spec each time one does. Adds `VectorQueryProto.covering_projection` (field 15) to reserve the tag, with the one initializer the new field forces on `query_to_proto`; no writer emits carried values yet, so the implementation follows separately.
vivek-bharathan
force-pushed
the
vb/covering-storage-format
branch
from
August 29, 2026 00:31
fb3db9d to
201a9e1
Compare
lance-gatekeeper
Bot
removed
the
K-changes
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#8535 declared
IndexMetadata.covering_fieldsbut left the physical side unspecified; this specifies it -- carried values are extra columns inauxiliary.idx, discovered by exclusion against the quantizer's internal columns and bound to their source fields by a newcovering_field_idsmetadata key, with noindex_versionbump.It also permits an index to carry a column it is also keyed on, the only case where an id repeats in
fields, so readers must take the carried set fromcovering_fieldsrather than subtract the keyed prefix.Adds
VectorQueryProto.covering_projection(field 15) to reserve the tag; no writer emits carried values yet, so the implementation follows separately.