Concurrent query execution: full integration (concurrency stack + late-mat + hardening tiers 1-5) - #1583

Draft

felipeblazing wants to merge 141 commits into

sirius-db:devfrom

felipeblazing:integration/concurrency-full

Commits on Aug 13, 2026

feat(scan): fused scan-filter — evaluate a scan's filter while decompressing

authored and
joosthooz
committed
ee89260
docs(compression): plan for intent-level decode-pushdown API and codegen dedup

joosthooz
and
claude
committed
be207fc
refactor(simpatico): dedup decode emitters and launchers, single-source kernel args

joosthooz
and
claude
committed
fd92979
refactor(simpatico): share the mask survivor loop across consuming variants

joosthooz
and
claude
committed
c9c3b72
docs(compression): note that #1380's APIs are now on dev

joosthooz
and
claude
committed
8506863
fix(scan): guard the null ingestible in sirius_gpu_scan_operator

joosthooz
and
claude
committed
0c585c1
refactor(simpatico): fold the delta masked store into emit_delta_producer

joosthooz
and
claude
committed
243b21e
refactor(simpatico): replace DecodeVariant with an enumerator x consumer shape

joosthooz
and
claude
committed
6cfde9d
docs(compression): record Part D as done and analyse growth on the shape axes

joosthooz
and
claude
committed
671e6e2
docs(compression): derive the candidate points on both shape axes

joosthooz
and
claude
committed
a6665dd
refactor(scan): carry the decode outcome as a value, not a dynamic type

joosthooz
and
claude
committed
99f6378
refactor(scan): declare the BOOL8 substitution instead of sniffing for it

joosthooz
and
claude
committed
aef5c6d
docs(compression): mark Phase 1 done and add a cold-start brief for Phase 2

joosthooz
and
claude
committed
d75f396
refactor(scan): put one facade between the scan and filtered decompression

joosthooz
and
claude
committed
5c3eeba
refactor(compression): name the decode machinery for what it does

joosthooz
and
claude
committed
79f09d2
docs(compression): record Phase 2 and hand off Phase 3

joosthooz
and
claude
committed
47c92f2
refactor(compression): one decode route, one column probe

joosthooz
and
claude
committed
c0bd2e2
refactor(compression): one list of row-selecting sources

joosthooz
and
claude
committed
110ded6
refactor(compression): decompress_scan_filter returns one finished table

joosthooz
and
claude
committed
e1e9183
refactor(compression): one definition per decode-policy knob

joosthooz
and
claude
committed
da061d3
docs(compression): close out Phase 3

joosthooz
and
claude
committed
63a1c9e
refactor(scan): the residual filter is a predicate, built once

joosthooz
and
claude
committed
a0815f4
feat(scan): drop conjuncts the decode already applied

joosthooz
and
claude
committed
1538893
docs(compression): record the AST residual and what elision still needs

joosthooz
and
claude
committed
adef8b9
refactor(scan): lower filter constants through the existing numeric_range

joosthooz
and
claude
committed
996500a
docs(compression): record why the decode knobs are still env vars

joosthooz
and
claude
committed
84302ac
test(compression): cover the selection wave's four device helpers

joosthooz
and
claude
committed
9fe2994
docs(compression): record why the mask kernels stay hand-written

joosthooz
and
claude
committed
cafd870
refactor(scan): share the filter-column resolution between the two walkers

joosthooz
and
claude
committed
f877649
fix(compression): key the thread-local stream pools by device

joosthooz
and
claude
committed
a295d6f
docs(compression): keep the orchestrator's capability checks, and say why

joosthooz
and
claude
committed
c4e0116
docs(compression): record the status, the measurements, and what was tried

joosthooz
and
claude
committed
f8c1347
refactor(compression): delete the pair-predicate machinery, it has no customer

joosthooz
and
claude
committed
a6cb5af
refactor(compression): drop two dead entry points and the prose that outlived its design

joosthooz
and
claude
committed
d4583b2
refactor(compression): the launcher surface is one entry point per consumer

joosthooz
and
claude
committed
b387da0
refactor(compression): name the policy after the feature, not the phase

joosthooz
and
claude
committed
116fa69
refactor(compression): one number for the chunk, one place for emitted CUDA

joosthooz
and
claude
committed
b05d79a
docs(compression): document filtering while decompressing where pins are chosen

joosthooz
and
claude
committed
2feb90e
modified task creator to be support concurrent queries

wmalpica
authored and
felipeblazing
committed
0e30048
made task_creator be able to handle multiple queries

wmalpica
authored and
felipeblazing
committed
2988763
fixed linter issue

wmalpica
authored and
felipeblazing
committed
8ed6b2d
improved request_keys_for

wmalpica
authored and
felipeblazing
committed
87d8519
added missing file

wmalpica
authored and
felipeblazing
committed
1d5b964
changed how completion_handler is stored and used. Removed _query and get_query_id() from task_scheduler. Removed _no_pref_rr_counter

wmalpica
authored and
felipeblazing
committed
70bd821
deleted SiriusContext::query_ and its usages

wmalpica
authored and
felipeblazing
committed
998222d
introducing query_scan_manager_state

wmalpica
authored and
felipeblazing
committed
b040ddf
fixed linter issues

wmalpica
authored and
felipeblazing
committed
394d09b
concurrency step 1: stop per-query failures from killing shared subsystems

authored and
felipeblazing
committed
8cc4143
cleaning up

wmalpica
authored and
felipeblazing
committed
be44a10
concurrency step 2: per-query lifecycle gate, and fix the C1 deadlock it exposed

authored and
felipeblazing
committed
48db5bc
linter fix

wmalpica
authored and
felipeblazing
committed
5da74a5

Commits on Aug 14, 2026

feat: late materialization v1-v3, rebased onto #1474's decode-pushdown API

felipeblazing
and
claude
committed
76906d4
Merge branch 'rb/concurrency3_0' into integration/concurrency-latemat
felipeblazing
committed
129b2cd
fix(test): follow scan_filter_status rename to declined_unselective

felipeblazing
and
claude
committed
459ce4f
docs(concurrency): record C1/C2 pull-forward into step 2

authored and
felipeblazing
committed
257709b
concurrency step 3: per-query completion and error paths

authored and
felipeblazing
committed
16f9777
fix(util): bound the crash handler so a segfault dies instead of spinning

authored and
felipeblazing
committed
55c9008
concurrency step 4: close the in-flight tracking holes and unstarve lookahead

authored and
felipeblazing
committed
0e3717d
concurrency step 5: query-aware thread pool with per-query waits

authored and
felipeblazing
committed
4f82baa
docs(concurrency): record steps 6+7 as attempted and backed out

authored and
felipeblazing
committed
6999fd5
concurrency steps 8+11: shutdown ordering, failed-cleanup leak, loud drops

authored and
felipeblazing
committed
825d712
concurrency steps 9+12 (partial): prefetch-cache UAF, real max_concurrent_queries

authored and
felipeblazing
committed
74fe656
docs(concurrency): execution summary and recommended next steps

authored and
felipeblazing
committed
ae4fe8d
feat(concurrency): counted query-lifecycle gate sized by max_concurrent_queries

felipeblazing
and
claude
committed
55462d4
fix(concurrency): serialize downgrade_executor drain/stop; harden concurrency test

felipeblazing
and
claude
committed
0eb35ad
fix(concurrency): per-query GPU admission on the shared task_creator

felipeblazing
and
claude
committed
c615f8c
fix(concurrency): error-path quiesce bracket no longer starves co-tenant queries

felipeblazing
and
claude
committed
3d38b18
fix(concurrency): admission may register the query state (window-less test drives)

felipeblazing
and
claude
committed
c90a8c7
fix(concurrency): serialize task_executor manager-thread lifecycle transitions

felipeblazing
and
claude
committed
635aef9
test(concurrency): shared-GPU-friendly small-pool default for the harness

felipeblazing
and
claude
committed
0b4aef5
test(concurrency): shared harness for the adversarial concurrency suite

felipeblazing
and
claude
committed
b72c05a
test(concurrency): adversarial workload scenarios

felipeblazing
and
claude
committed
e70b020
test(concurrency): group-E config-race adversarial scenarios

felipeblazing
and
claude
committed
b4fab0e
test(concurrency): mixed failure storm goes live as a regression guard

felipeblazing
and
claude
committed
14fe561
test(concurrency): appease codespell in the config-races header comment

felipeblazing
and
claude
committed
72ad8e2
fix(concurrency): TIER-2 re-push uses extraction-time keys, not a re-derived plan walk (B1)

felipeblazing
and
claude
committed
ef92806
fix(concurrency): destroy the plan AFTER the query's drains, not before (B5)

felipeblazing
and
claude
committed
562ca50
test(concurrency): teardown-vs-downgrade race harness (B1/B5 interlock)

felipeblazing
and
claude
committed
be757a4
docs(concurrency): record B1/B5 fixes (and the resume_manager fix) in the bring-up triage

felipeblazing
and
claude
committed
7325961
feat(concurrency): sweep gate fences repository teardown against downgrade sweeps

felipeblazing
and
claude
committed
9cd5fe8
fix(concurrency): per-query downgrade drain; monitor re-arms after cancelled request

felipeblazing
and
claude
committed
7772e46
test(concurrency): per-query drain semantics, monitor re-arm, query-end churn under pressure

felipeblazing
and
claude
committed
3ca2781
fix(concurrency): interlock per-query downgrade drain with parked-plan destruction

felipeblazing
and
claude
committed
e51248e
fix(concurrency): D5 — contain a query's cleanup failure to that query

felipeblazing
and
claude
committed
6e10465
fix(concurrency): B9 — locked shared_ptr accessor for data_repository_manager lookups

felipeblazing
and
claude
committed
4a045bc
fix(concurrency): round-robin queue pops across query bands (F1)

felipeblazing
and
claude
committed
39730f6
test(concurrency): cross-query fairness coverage for the F1 fix

felipeblazing
and
claude
committed
950979a
docs(concurrency): record the F1 fairness fix in the bring-up triage

felipeblazing
and
claude
committed
aef3b61
fix(concurrency): E1/E2 snapshot-at-window-begin for SET-mutable config

felipeblazing
and
claude
committed
effa4ee
fix(concurrency): E4 immutable shared transparent prepared operator

felipeblazing
and
claude
committed
59fb9a9
fix(concurrency): E7 per-execution ClientConfig save/restore in gpu_execution

felipeblazing
and
claude
committed
7274fb0
test(concurrency): green E4 engagement assertions, SET storm, E7 restore

felipeblazing
and
claude
committed
73c609a
test(config): deterministic unit test for the query config snapshot

felipeblazing
and
claude
committed
4e8d0c3
docs(concurrency): record the E1/E2/E4/E7 fixes in the bring-up triage

felipeblazing
and
claude
committed
7a734f2

Commits on Aug 15, 2026

test(concurrency): the SET storm restores every knob it perturbs on exit

felipeblazing
and
claude
committed
36191e3
fix(concurrency): E3 compression config rides the admission snapshot

felipeblazing
and
claude
committed
a13b04b
fix(concurrency): E5 identity-keyed atomic compression plan registry

felipeblazing
and
claude
committed
0f7b252
fix(concurrency): E6 per-device atomic cache in get_target_ctas

felipeblazing
and
claude
committed
0e4b4e1
test(concurrency): E3/E5/E6 regression coverage

felipeblazing
and
claude
committed
3ce4cac
docs(concurrency): record the E3/E5/E6 fixes — Group E is closed

felipeblazing
and
claude
committed
e4991c6
feat(exec): make the GPU OOM-retry cap a per-query config knob

felipeblazing
and
claude
committed
9cf2af8
fix(extension): honor gpu_execution's enable_optimizer named parameter

felipeblazing
and
claude
committed
0c515da
feat(transparent): intercept SQL PREPARE/EXECUTE for parameterless SELECTs

felipeblazing
and
claude
committed
9396ef1
docs(concurrency): record the retry-cap knob, enable_optimizer and PREPARE/EXECUTE fixes

felipeblazing
and
claude
committed
e346986
fix(concurrency): H8 — guard the window-id allocator across the 2^32 wrap

felipeblazing
and
claude
committed
b7103ab
chore(concurrency): H-group sirius-side dead-code deletions and comment drift

felipeblazing
and
claude
committed
4f6d974
chore(concurrency): bump cucascade for the H-group hygiene commit

felipeblazing
and
claude
committed
caf75d8
fix(io/cache): F3/F4 — cross-query fairness in the prefetching cache

felipeblazing
and
claude
committed
1546678
test(io/cache): cross-query fairness coverage for the prefetch cache

felipeblazing
and
claude
committed
0d2a6ad
docs(concurrency): record the F3/F4 prefetch-cache fairness fixes

felipeblazing
and
claude
committed
423a78a
fix(io/cache): B11 stragglers — device_read_async used the map iterator after unlock

felipeblazing
and
claude
committed
dbf4b82
test(concurrency): per-PID temp files for concurrent_env

felipeblazing
and
claude
committed
f194d88
fix(concurrency): C4 — the GPU manager thread never blocks on memory

felipeblazing
and
claude
committed
cf4b311
fix(concurrency): D3 — lookahead rotates across queries and is teardown-safe

felipeblazing
and
claude
committed
38338a6
docs(concurrency): record the C4 and D3 fixes in the bring-up triage

felipeblazing
and
claude
committed
9061821
fix(telemetry): A8 — query end drains only that query's batch placements

felipeblazing
and
claude
committed
903e2bd
fix(concurrency): F2 — plan generation no longer occupies an execution slot

felipeblazing
and
claude
committed
f1b76a8
docs(concurrency): SlotGuard header comment no longer claims plan windows

felipeblazing
and
claude
committed
48cfa0f
fix(scan_manager): F5 — match snapshots reference only entries that can serve

felipeblazing
and
claude
committed
b01fe55
fix(scan_manager): a pin name may only be re-used for the same resolved identity

felipeblazing
and
claude
committed
9c766a5
docs(concurrency): record the A8/F2/F5 and pin-name identity fixes in the triage

felipeblazing
and
claude
committed
a16e8f8
fix(concurrency): F6 — size sort partitions and collector host space per query, not per device

felipeblazing
and
claude
committed
fa5c5bc
fix(concurrency): F7 — remove device-wide syncs and legacy default-stream work from operators

felipeblazing
and
claude
committed
14e0418
test(concurrency): F6 sizing scenarios — concurrent sorts and wide-result collectors

felipeblazing
and
claude
committed
4d6dc6d
docs(concurrency): record the F6/F7 fixes in the bring-up triage

felipeblazing
and
claude
committed
1e080f3
docs(super-sirius): fix post-concurrency drift — deleted APIs, per-query lifecycle

felipeblazing
and
claude
committed
157827e
docs(super-sirius): add the per-query concurrency model doc

felipeblazing
and
claude
committed
a19769e
docs(concurrency): STATUS addendum on the execution summary

felipeblazing
and
claude
committed
482fcdc
docs(super-sirius): bump last-updated-commit marker to 90618210

felipeblazing
and
claude
committed
b109f7c
docs(super-sirius): refresh the SCHED-RR contract snippets to the current code

felipeblazing
and
claude
committed
b81b927
fix(concurrency): F9 — FIFO wait discipline in cucascade's shared blocking primitives

felipeblazing
and
claude
committed
21067be
fix(concurrency): the error bracket waits per-query, not for the whole pool

felipeblazing
and
claude
committed
73ba755
test(concurrency): F9 fairness and per-query error-bracket coverage

felipeblazing
and
claude
committed
f996829
docs(concurrency): record the F9 fix in the bring-up triage

felipeblazing
and
claude
committed
a1191f8
feat(concurrency): step 6 — shared-ownership data repositories end-to-end

felipeblazing
and
claude
committed
37bf952
feat(concurrency): step 7 prep — retire the sweep gate; the plan-lifetime fence survives

felipeblazing
and
claude
committed
dae6241
feat(concurrency): step 7 — delete the global downgrade drain

felipeblazing
and
claude
committed
22d4ca5
feat(concurrency): F8 — concurrent downgrade requests with peers-first victim selection

felipeblazing
and
claude
committed
c3005ce
feat(concurrency): D5 step-level classification inside the mandatory cleanup

felipeblazing
and
claude
committed
9b6b40a
docs(concurrency): record steps 6+7, F8 and step-level D5 in the bring-up triage

felipeblazing
and
claude
committed
4a2c7b7
chore(concurrency): codespell fix in the downgrade executor header

felipeblazing
and
claude
committed
2c59e32