[VQueues] Optimize sys_vqueue_entry_status point lookups#5047
Open
AhmedSoliman wants to merge 3 commits into
Open
[VQueues] Optimize sys_vqueue_entry_status point lookups#5047AhmedSoliman wants to merge 3 commits into
AhmedSoliman wants to merge 3 commits into
Conversation
This was referenced Jul 15, 2026
Test Results 8 files 8 suites 4m 59s ⏱️ Results for commit cf47df9. ♻️ This comment has been updated with latest results. |
AhmedSoliman
force-pushed
the
pr5047
branch
2 times, most recently
from
July 16, 2026 15:59
e6a56c5 to
709347d
Compare
AhmedSoliman
marked this pull request as ready for review
July 16, 2026 16:07
Replace the specialized partition and key-range scan variants with a compact TableScan API that lowers encoded bounds to prefix or exclusive range scans. Use prefix seeking only when the encoded bounds remain within RocksDB's fixed prefix, fall back to total-order seeking for cross-prefix ranges, and turn singleton ranges into prefix scans. Centralize iterator-bound configuration across direct, background, and transactional readers. Let callers supply ReadOptions so async I/O and transaction snapshots remain caller policy, while short prefixes safely bypass the fixed-prefix extractor and its bloom filters. Remove the iterator wrapper plumbing, share encoded Bytes bounds across RocksDB paths, and migrate table and storage-migration callers to the consolidated API. Cover inclusive upper bounds, singleton partition ranges, and prefix-versus-total-order selection.
… reads Keep equality and IN predicates as exact ID selections in the DataFusion filter layer instead of collapsing large sets into scan ranges. This lets point-lookup tables preserve the user's requested IDs while range-scan tables can still convert the same selection back to bounds. Teach the partitioned table provider to group exact point reads by Restate partition when the scanner can re-fetch the exact IDs itself. Use that for sys_invocation_status by adding an InvocationIdSet scan path backed by bounded RocksDB multi-get batches. Split partition-store inclusive key-range scans into single-prefix and total-order variants, and use total-order scans when invocation-status, journal, or journal-events ID bounds span multiple partition keys. This preserves correctness for range-scan consumers of exact ID selections. Adapt the worker cleaner to the fallible stream shape used by the new invocation-status scan path, and add scan-mode coverage for the new total-order range variant. This design delivers 5-10x better performance for point-lookups than the concurrent iterator-per-entry approach that was previously used. This pays more dividends on slower disks and will allow us to leverage async I/O better when we build rocksdb with folly.
Make sys_vqueue_entry_status understand exact entry_id predicates so equality and IN filters can be planned as targeted reads rather than broad partition scans. Entry IDs are parsed back to their partition keys for routing and displayed consistently from the stored EntryId representation. Add the partition-store support needed to read selected entry-status records through bounded batched multi-get calls while preserving range scans for non-point predicates. Entry-id range scans now use total-order iteration when the bounds span multiple partition keys. Only advertise partition_key ordering for sys_vqueue_entry_status, because the exposed entry_id string ordering does not match the raw on-disk entry-id ordering. Cover both the DataFusion point-query path and multi-get batching behavior, including a NOT IN regression so excluded IDs are not treated as an exact lookup set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make sys_vqueue_entry_status understand exact entry_id predicates so equality and IN filters can be planned as targeted reads rather than broad partition scans. Entry IDs are parsed back to their partition keys for routing and displayed consistently from the stored EntryId representation.
Add the partition-store support needed to read selected entry-status records through bounded batched multi-get calls while preserving range scans for non-point predicates. Entry-id range scans now use total-order iteration when the bounds span multiple partition keys.
Only advertise partition_key ordering for sys_vqueue_entry_status, because the exposed entry_id string ordering does not match the raw on-disk entry-id ordering.
Cover both the DataFusion point-query path and multi-get batching behavior, including a NOT IN regression so excluded IDs are not treated as an exact lookup set.
Stack created with Sapling. Best reviewed with ReviewStack.