KAFKA-20705: IQv2 TimestampedWindowKeyWithHeadersQuery for headers-aware window stores (KIP-1356)#22799
Draft
Jess668 wants to merge 18 commits into
Draft
Conversation
- Add the headers-aware range query TimestampedRangeWithHeadersQuery, returning a ReadOnlyRecordIterator of ReadOnlyRecord (key/value/timestamp/headers). - MeteredTimestampedKeyValueStoreWithHeaders handles it by forwarding a raw RangeQuery to the wrapped store and adapting each entry into a frozen-header ReadOnlyRecord; an entry whose stored timestamp is negative cannot be represented as a ReadOnlyRecord, so advancing the iterator throws StreamsException. - Also drop the query() override on RocksDBTimestampedStoreWithHeaders so the native header store serves the basic IQv2 queries via the inherited RocksDBStore handling.
- TimestampedKeyValueStoreBuilderWithHeadersTest: end-to-end build-path coverage (native/in-memory return headers; adapter returns empty headers; plain-legacy and negative timestamps throw while iterating), execution-info collection, and native/adapter parity. - MeteredTimestampedKeyValueStoreWithHeadersTest: unit coverage of the metered handler forwarding the bounds and result order onto the raw RangeQuery. - RocksDBTimestampedStoreWithHeadersTest: the native store now serves RangeQuery. - TimestampedRangeWithHeadersQueryIntegrationTest: IQv2 integration coverage.
…ntion - Query with PositionBound.at(INPUT_POSITION) + iqv2WaitForResult
Add regression tests for the num-open-iterators gauge (0->1->0, misuse paths of double-close and unclosed-after-throw), the raw-query failure propagation branch, and the missing bound/order capture combinations (no-bounds/ANY, both-bounds/ANY) in the metered store's IQv2 handling.
- Add a stored-record assertion to shouldHandleRangeQuery - Strengthen the native/adapter range parity test to use multiple keys - Assert exception messages (not just type) for the negative-timestamp range failures
Add integration test cases: caching-enabled range query bypassing the cache, a descending bounded range, equal and inverted range bounds, an empty-range no-match case, and an ADAPTER-config range query asserting empty (not null) headers. Rename the KeyValue-store-specific processors and helpers (writer processors, query/querySkipCache, startStreamsWith* setup methods) with a KeyValue prefix/suffix, since this class is documented to be extended by window/session store query types as they land and the unqualified names would become ambiguous.
- Introduce the headers-aware windowed point query, returning a ReadOnlyRecordIterator<Windowed<K>, V> whose elements carry the stored record headers. - MeteredTimestampedWindowStoreWithHeaders handles it by forwarding a raw WindowKeyQuery to the wrapped store and wrapping each entry as a ReadOnlyRecord (Windowed key + value + timestamp + frozen headers). - The native header window stores (RocksDBTimestampedWindowStoreWithHeaders, RocksDBTimeOrderedWindowStoreWithHeaders) special-case WindowKeyQuery to super.query() so it is served via StoreQueryUtils; every other query type (incl. WindowRangeQuery) stays UNKNOWN_QUERY_TYPE, deferred to a follow-up, mirroring the KV KeyQuery PR. - Also fixes a latent WindowKeyQuery constructor bug that swapped timeFrom/timeTo.
c125059 to
e932a4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
TimestampedWindowKeyWithHeadersQuery, the third IQv2 query type of KIP-1356 — the headers-aware parallel ofWindowKeyQuery. Its result is an iterator of windowed records that each carry the windowed key, value, timestamp (record event-time), and headers. The result type isReadOnlyRecordIterator<Windowed<K>, V>(already intrunk) — aCloseableiterator that yieldsReadOnlyRecord<Windowed<K>, V>elements directly.Dependencies
This work is stacked on, and should be reviewed/merged after:
ReadOnlyRecordPR#22677 — already intrunk.ReadOnlyRecordIteratorPR#22729 — already intrunk.TimestampedKeyWithHeadersQueryPR#22666 — already intrunk.TimestampedRangeWithHeadersQueryPR#22770 — the preceding KIP-1356 query type; this branch is stacked on it and should be rebased ontotrunkonce it lands, leaving only the window-key commits.Summary
TimestampedWindowKeyWithHeadersQuery<K, V> implements Query<ReadOnlyRecordIterator<Windowed<K>, V>>(
@Evolving/@InterfaceAudience.Public). ParallelsWindowKeyQuery— single static factorywithKeyAndWindowStartRange(key, timeFrom, timeTo), a closed window-start range is required — but usesthe newer no-
getaccessors (key(),timeFrom(),timeTo()) and rejects a null key, consistentwith
TimestampedKeyWithHeadersQuery/TimestampedRangeWithHeadersQuery. Each element's key is aWindowed<K>(the queried key + the entry's window);timestamp()is the stored record event-time.MeteredTimestampedWindowStoreWithHeadershandles the new query type by forwarding a rawWindowKeyQueryto the wrapped store and wrapping the returnedWindowStoreIteratorin aReadOnlyRecordIteratorwhose elements are immutableReadOnlyRecords (headers frozen viasetReadOnly()). An entry whose stored timestamp is negative cannot be represented as aReadOnlyRecord, so the lazily-evaluated iterator throwsStreamsExceptionwhile advancing.RocksDBTimestampedWindowStoreWithHeadersandRocksDBTimeOrderedWindowStoreWithHeadersspecial-caseWindowKeyQueryto the inherited storehandling (
StoreQueryUtils) viasuper.query(); previously they returnedUNKNOWN_QUERY_TYPEforevery query. Every other query type — including
WindowRangeQuery— remainsUNKNOWN_QUERY_TYPEandis deferred to a follow-up, mirroring how the
TimestampedKeyWithHeadersQueryPR special-casedKeyQuery(rather than dropping the override outright).WindowKeyQueryconstructor fix: its private constructor declared its parameters as(key, timeTo, timeFrom)while thewithKeyAndWindowStartRangefactory passes(key, timeFrom, timeTo), sogetTimeFrom()/getTimeTo()returned each other's values. This wasmasked in the existing
MeteredWindowStorepath (which re-wraps the query through the same factory,cancelling the swap) but is exposed by the new handler and the now-enabled native path, both of which
read the accessors once. Fixed the parameter order; verified net-invariant for the existing path
(
StoreQueryUtilsTest,MeteredWindowStoreTest,IQv2StoreIntegrationTest).Testing
MeteredTimestampedWindowStoreWithHeadersTest: the typed query is translated into a rawWindowKeyQuerycarrying the serialized key and the window-start bounds.TimestampedWindowStoreWithHeadersBuilderTest(QueryTests):TimestampedWindowKeyWithHeadersQueryreturns exact headers on a header-persisting (native) store,empty headers on the adapter store, throws on a plain supplier and on a negative stored timestamp;
execution-info collect/no-collect; identical results across native- and adapter-built stores.
RocksDBTimestampedWindowStoreWithHeadersTest/RocksDBTimeOrderedWindowStoreWithHeadersTest:WindowKeyQueryis now handled (wasUNKNOWN_QUERY_TYPE);WindowRangeQueryremainsUNKNOWN_QUERY_TYPE.IQv2HeadersStoreIntegrationTest:TimestampedWindowKeyWithHeadersQueryover arunning topology returns each record's windowed key/value/timestamp/headers across a window-start
range (empty headers and read-only headers included, tombstoned window omitted);
UNKNOWN_QUERY_TYPEagainst a non-headers store; failure (throws while iterating) against a plain supplier.