perf(jobs): job-burst gate bench + static-headroom guidance; don't autoscale the job pool (#323 phase 3)#329
Conversation
…toscale the job pool (#323 phase 3) Issue #323 phase 3 mandates measuring which pool a job burst starves before building elastic job pools. benchmarks/job_burst_bench.rb (rake bench:job_burst) sweeps worker thread count under a fixed burst in two connection regimes and measures throughput + peak_busy via the shared ExecutionModeHarness. MEASURED (real DB): a job holds a pgmq connection only for the read_batch + archive round-trip, not the job body — so the DB connection pool is the hard burst ceiling. In the fixed-pool regime peak_busy never exceeds pool_size no matter how many threads run (extra threads queue on checkout; p99 latency climbs 43ms → 227ms; throughput plateaus). Matching the pool to the thread count scales throughput ~linearly (8× from 2→16). CONCLUSION (issue #323's sanctioned outcome): elastic job threads on a fixed connection pool cannot push past the connection ceiling, so pgbus does NOT autoscale the job execution pool. The fix for job bursts is static headroom — raise threads (pool_size follows, or keep it ≥ threads). This is why the design recommended measuring first: it saved building a feature that wouldn't work. Docs: new "Job bursts: raise threads and the pool together" section in the Performance & tuning page; bench added to the runbook + CHANGELOG. The streams pool autoscaling stays elastic where it genuinely pays off (its connection cost scales with its size). No production code changed.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
mhenrixon
added
documentation
1 participant
Summary
Issue #323 phase 3 (elastic job execution pools) mandates a measure-first gate: confirm which pool a job burst actually starves before building elasticity, and — the issue's words — "if static headroom already suffices, close as 'static headroom is the answer.'" This PR runs that gate and reaches exactly that conclusion, with data.
No production code changes — a benchmark, operator guidance, and a decision.
The measurement
benchmarks/job_burst_bench.rb(rake bench:job_burst) sweeps worker thread count under a fixed 400-job burst in two connection regimes, using the sharedExecutionModeHarness(the same rig as #324):pool_size == threads(the "raise both" static-headroom answer)pool_sizefixed at 4 while threads climb (the "elastic threads, fixed connection pool" case — exactly what elastic job threads without elastic connections would produce)Measured on a real DB (db-bound jobs, 30ms in-checkout each):
The finding
A job holds a pgmq connection only for the brief
read_batch+archiveround-trip — not for the job body (confirmed atclient.rb:83, where the job pool is a plain fixedPGMQ::Client). So the DB connection pool is the hard burst ceiling, not the thread count. In the STARVED regime,peak_busynever exceedspool_sizeno matter how many threads run — the extra threads just queue on connection checkout (p99 latency climbs 43ms → 227ms, throughput flatlines). Matching the pool to the thread count scales throughput ~linearly.The decision
pgbus does NOT autoscale the job execution pool — issue #323's explicitly sanctioned outcome. Elastic job threads on a fixed connection pool cannot push past the connection ceiling, so the machinery wouldn't move throughput. The fix for job bursts is static headroom: raise
threads(and letpool_sizefollow, or keep it ≥threads). Idle pool slots are lazy — they cost nothing until a burst uses them.This is why the design pass recommended measuring first: it saved building a feature that wouldn't work. (It also found — and reproduced against installed concurrent-ruby — a permanent capacity-drift double-run bug in the more ambitious "control-loop" design variant, a second reason not to go there. The rebuild-and-swap primitive from that variant is the right foundation for phase 4's streams/writer pools, not here.)
The streams-pool autoscaling (#327/#328) stays the elastic story where it genuinely pays off — the streams pool's connection cost does scale with its size.
Changed
benchmarks/job_burst_bench.rb+rake bench:job_burst(run-and-report, never a CI gate)lib/changes.Test plan
bundle exec rubocopcleanrake bench:job_burstruns against a real DB and prints the verdict aboverake lintclean, 57 docs specs greenDeviations & judgment calls
peak_busycapping atpool_size, not a raw throughput ratio. A first cut used a throughput-plateau threshold, which the db-bound profile's tiny non-checkout slice tripped (a couple extra threads overlap it for a one-time ~20% bump before flatlining). The honest signal that threads-past-the-pool do no concurrent work ispeak_busy ≤ pool_size— so the verdict reads that.ThreadPoolExecutor(min/max)) is fully designed but deliberately not built — the gate says it wouldn't help. If a future workload with a genuinely thread-bound job profile appears, the design is ready in the issue history.Refs #323.