Skip to content

fix(label_list): bound memory in index build, update, and merge - #8840

Open
vivek-bharathan wants to merge 1 commit into
lance-format:mainfrom
vivek-bharathan:vb/labellist-spilling
Open

fix(label_list): bound memory in index build, update, and merge#8840
vivek-bharathan wants to merge 1 commit into
lance-format:mainfrom
vivek-bharathan:vb/labellist-spilling

Conversation

@vivek-bharathan

@vivek-bharathan vivek-bharathan commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

LabelList materialized its whole label-to-bitmap map in memory during build, update, merge and remap, so peak memory scaled with distinct-label cardinality and large list columns led to out of memory errors.

Build and update now accumulate into a byte-budgeted sorted map (LANCE_LABEL_LIST_SPILL_BYTES, default 512 MiB), spill sorted files to local scratch and k-way merge them; update rewrites the existing index into one more sorted scratch file so it joins the same merge.

This also changes the BITMAP index. The shared bitmap writer's flush threshold drops from ~2 GiB to 32 MiB and now charges keys as well as serialized bitmaps, fixing a latent overflow where a high-cardinality column with tiny bitmaps could exceed the keys column's i32 offsets before ever tripping a flush.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Aug 28, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 28, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 28, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 29, 2026
LabelList materialized its whole label-to-bitmap map in memory during
build, update, merge and remap, so peak memory scaled with distinct-label
cardinality and large list columns led to out of memory errors.

Build and update now accumulate into a byte-budgeted sorted map
(LANCE_LABEL_LIST_SPILL_BYTES, default 512 MiB), spill sorted files to
local scratch and k-way merge them; update rewrites the existing index
into one more sorted scratch file so it joins the same merge.

This also changes the BITMAP index. The shared bitmap writer's flush
threshold drops from ~2 GiB to 32 MiB and now charges keys as well as
serialized bitmaps, fixing a latent overflow where a high-cardinality
column with tiny bitmaps could exceed the keys column's i32 offsets
before ever tripping a flush.
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 29, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The rebase leaves the spill-backed implementation unchanged and preserves LabelList v1 behavior across forced spills, legacy unsorted files, nulls, remap, updates, and filtered segment merge.

The non-blocking risk remains: LANCE_LABEL_LIST_SPILL_BYTES bounds only estimated mutable aggregation state, not total process memory. Large postings or cursor batches, list-null state, scan batches, caches, and output buffering remain outside it, so operators should not treat the setting as a hard RSS ceiling.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 29, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 29, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant