Skip to content

KAFKA-20597: Fix prefixScan overflow for prefixes with no upper bound in in-memory state stores#22789

Open
edenfunf wants to merge 2 commits into
apache:trunkfrom
edenfunf:KAFKA-20597-prefixscan-overflow
Open

KAFKA-20597: Fix prefixScan overflow for prefixes with no upper bound in in-memory state stores#22789
edenfunf wants to merge 2 commits into
apache:trunkfrom
edenfunf:KAFKA-20597-prefixscan-overflow

Conversation

@edenfunf

@edenfunf edenfunf commented Jul 9, 2026

Copy link
Copy Markdown

Problem

When a prefix has no lexicographically larger successor (e.g. it is all 0xFF bytes), prefixScan computes the exclusive upper bound of its range by incrementing the prefix, which overflows. RocksDBStore already handles this by treating the overflow as an unbounded range, but InMemoryKeyValueStore, MemoryNavigableLRUCache and CachingKeyValueStore called ByteUtils.increment() directly and threw IndexOutOfBoundsException.

Fix

  • Centralize the overflow-safe increment as ByteUtils.incrementWithoutOverflow(), which returns null on overflow. The RocksDB code paths that previously defined a private copy now reuse it.
  • The three in-memory stores treat a null upper bound as "scan to the end of the keyspace" (tailMap), matching the behaviour RocksDBStore already had.
  • ByteUtils.increment() is left unchanged: RocksDBStore.deleteRange() intentionally relies on it throwing, since RocksDB's deleteRange() has no null-upper-bound form.

Testing

  • Added prefixScanShouldReturnMatchesWhenPrefixHasNoUpperBound to AbstractKeyValueStoreTest, so it runs against every KeyValueStore implementation. It uses IntegerSerializer's encoding of -1 (0xFFFFFFFF) to reproduce the overflow. Verified the test fails on the unpatched in-memory stores and passes with the fix, while the RocksDB-backed stores pass in both cases.
  • Added unit tests for ByteUtils.incrementWithoutOverflow().

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

… in in-memory state stores

When a prefix has no lexicographically larger successor (e.g. it is all
0xFF bytes), incrementing it to compute the exclusive upper bound of the
scan range overflows. RocksDBStore already handled this by treating the
overflow as an unbounded range, but InMemoryKeyValueStore,
MemoryNavigableLRUCache and CachingKeyValueStore called
ByteUtils.increment() directly and threw IndexOutOfBoundsException.

Centralize the overflow-safe increment as ByteUtils.incrementWithoutOverflow(),
which returns null on overflow, and reuse it from the RocksDB code paths that
previously defined a private copy. The in-memory stores now treat a null upper
bound as "scan to the end of the keyspace" (tailMap), matching RocksDB.

Add a regression test to AbstractKeyValueStoreTest so it runs against every
KeyValueStore implementation, using IntegerSerializer's encoding of -1
(0xFFFFFFFF) to reproduce the overflow, plus unit tests for
ByteUtils.incrementWithoutOverflow().
@github-actions github-actions Bot added triage PRs from the community streams clients labels Jul 9, 2026
@github-actions github-actions Bot added the small Small PRs label Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clients small Small PRs streams triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant