fix(label_list): bound memory in index build, update, and merge - #8840
Open
vivek-bharathan wants to merge 1 commit into
Open
fix(label_list): bound memory in index build, update, and merge#8840vivek-bharathan wants to merge 1 commit into
vivek-bharathan wants to merge 1 commit into
Conversation
vivek-bharathan
force-pushed
the
vb/labellist-spilling
branch
from
August 28, 2026 18:24
aa6cc44 to
c806b8e
Compare
LabelList materialized its whole label-to-bitmap map in memory during build, update, merge and remap, so peak memory scaled with distinct-label cardinality and large list columns led to out of memory errors. Build and update now accumulate into a byte-budgeted sorted map (LANCE_LABEL_LIST_SPILL_BYTES, default 512 MiB), spill sorted files to local scratch and k-way merge them; update rewrites the existing index into one more sorted scratch file so it joins the same merge. This also changes the BITMAP index. The shared bitmap writer's flush threshold drops from ~2 GiB to 32 MiB and now charges keys as well as serialized bitmaps, fixing a latent overflow where a high-cardinality column with tiny bitmaps could exceed the keys column's i32 offsets before ever tripping a flush.
vivek-bharathan
force-pushed
the
vb/labellist-spilling
branch
from
August 29, 2026 01:53
c806b8e to
f4a4ba8
Compare
Contributor
There was a problem hiding this comment.
The rebase leaves the spill-backed implementation unchanged and preserves LabelList v1 behavior across forced spills, legacy unsorted files, nulls, remap, updates, and filtered segment merge.
The non-blocking risk remains: LANCE_LABEL_LIST_SPILL_BYTES bounds only estimated mutable aggregation state, not total process memory. Large postings or cursor batches, list-null state, scan batches, caches, and output buffering remain outside it, so operators should not treat the setting as a hard RSS ceiling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LabelList materialized its whole label-to-bitmap map in memory during build, update, merge and remap, so peak memory scaled with distinct-label cardinality and large list columns led to out of memory errors.
Build and update now accumulate into a byte-budgeted sorted map (LANCE_LABEL_LIST_SPILL_BYTES, default 512 MiB), spill sorted files to local scratch and k-way merge them; update rewrites the existing index into one more sorted scratch file so it joins the same merge.
This also changes the BITMAP index. The shared bitmap writer's flush threshold drops from ~2 GiB to 32 MiB and now charges keys as well as serialized bitmaps, fixing a latent overflow where a high-cardinality column with tiny bitmaps could exceed the keys column's i32 offsets before ever tripping a flush.