diff --git a/CHANGELOG.md b/CHANGELOG.md index ececd65..da7357c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ ### Added +- **Job-burst tuning guidance + gate benchmark (`rake bench:job_burst`), and a decision NOT to autoscale the job pool (issue #323 phase 3).** Following the issue's measure-first mandate, `benchmarks/job_burst_bench.rb` sweeps worker thread count under a job burst in two connection regimes (pool matched to threads vs. pool fixed small). Measured finding: a job holds a pgmq connection only for the `read_batch` + `archive` round-trip (not the job body), so **the DB connection pool is the hard burst ceiling, not the thread count** — in the fixed-pool regime `peak_busy` never exceeds `pool_size` no matter how many threads run (extra threads just queue on checkout, latency climbs, throughput plateaus), while matching the pool to the thread count scales throughput ~linearly (≈8× from 2→16). The conclusion — issue #323's explicitly sanctioned outcome — is that **elastic job threads on a fixed connection pool cannot help**, so pgbus does *not* autoscale the job execution pool; the fix for job bursts is **static headroom**: raise `threads` (and let `pool_size` follow, or keep it ≥ `threads`). Documented in the Performance & tuning docs. The streams-pool autoscaling below stays the elastic story where it *does* pay off (the streams pool's connection cost genuinely scales with its size). Refs #323. - **Self-tuning elastic streams-pool autoscaling (`config.streams_pool_autoscale`, opt-in, default off).** The dedicated streams pool (durable-broadcast publish + dispatcher replay reads) is a fixed `streams_pool_size` (default 5); under a genuine burst of SSE clients it saturates, and a saturated pool *serialises* replay reads (it doesn't error — the checkout waits; see #324), slowing fan-out. Setting `config.streams_pool_autoscale = true` runs a **periodic maintenance check** (default every 5 minutes, `config.streams_pool_autoscale_interval`) that **grows the streams pool into a fair share of live Postgres connection headroom** while it's saturated and **shrinks it back to `streams_pool_size`** when the burst passes — with no connection-count target to tune: every threshold derives from live `max_connections`. The check adds **no extra connection and no extra thread** — it's a lightweight `pg_stat_activity` query that runs on the streamer's *existing idle LISTEN connection* (a pghero-`capture_query_stats`-style periodic task, run on the listener thread in its health-check window). It counts peer stream processes by a per-process `application_name` tag (`config.streams_application_name`, default `"pgbus_streams"`), and each process grows only into its own fair share of free connections (`free / peers`, claiming ¼, bounded by a per-grow step and half the remaining headroom) so N forked processes provably can't collectively exhaust the database — even in a synchronized cold-boot herd. If the database runs *critically* low on free connections, every process **emergency-shrinks to baseline immediately**, overriding the saturation signal (protecting the DB wins). `config.streams_pool_max` is an **optional** hard per-process ceiling (`nil` = the dynamic fair share is the cap). No per-sample hysteresis or cooldown is needed — the slow cadence is itself the debounce: one grow (or shrink) step per check, and a sustained burst converges over a few checks. Built on the #325 hot-swap primitive; the decision object is a pure function (no thread, no connection), DB-free unit-tested (scripted headroom, incl. convergence-across-checks and emergency-shrink), with a DB-gated integration test proving real grow/shrink/emergency and a `rake bench:pool_autoscale` benchmark (check cost ~0.3 ms; grow-under-saturation `3→7→10`). **Two prerequisite fixes ship with it:** streams-pool connections are now tagged with a per-process `application_name`, and the streams pool is now built from `streams_connection_options` (honoring `streams_database_url`/host/port) rather than the job-database options — the latter a latent wrong-DB bug in the existing streams pool that only surfaced with a separate streams database. Default-off = byte-identical (the Listener runs no maintenance). A pure-*publisher* worker process (no streamer, only `send_stream_message`) autoscales too: each publish opportunistically triggers the same check, throttled to at most once per `streams_pool_autoscale_interval` across all publisher threads (a lock-free compare-and-set claims the window). The publish thread does only that CAS and then hands the work to a background single-worker executor — it never runs the `pg_stat_activity` query or the resize inline (off the hot path), and a quiet process spawns no thread. The query runs through the *job* pool (so it never competes with the streams pool it grows), and the publisher decision is **grow-only**: it grows under its own publish pressure and emergency-shrinks on DB exhaustion, but leaves normal idle shrink to the streamer/consumer that sees the sustained-idle picture. Fail-soft throughout, so a broadcast never fails on a telemetry query. For steady load, a larger static `streams_pool_size` remains the simpler choice — autoscaling earns its keep under genuinely bursty streams traffic. `PoolAutoscaler` lives in `Pgbus::Streams::` (it is a streams-pool controller, not web-specific). Refs #323. - **Execution-mode connection benchmark + DB-pool sizing guidance (`rake bench:execution_modes`).** A fair, reusable harness (`benchmarks/support/execution_mode_harness.rb` + `benchmarks/execution_modes_bench.rb`) that runs the *same* offered load through the threads and async execution pools and measures how many real Postgres connections each actually holds (`peak_busy = pool size − available`, sampled through the real pool checkout), so operators can size the DB pool per `execution_mode`. It settles the "async saves connections" question with data: a job holds a pgmq connection only for the `read_batch` + `archive` round-trip (not the job body), so **async's connection-density win is real only for I/O-light work** (a few connections serve many fibers) — for **DB-bound work async is connection-bound like threads**, and a too-small pool *serialises* (throughput collapse, ~3.4× in the measured cell) rather than erroring, because `pool_timeout` (5 s) dwarfs a typical checkout. New `docs/performance.md` section "Execution mode and DB pool sizing" carries the measured numbers and a sizing table. The harness's fairness/struct/measurement logic is unit-tested with fakes (no DB, runs in CI); the numbers come from a manual `PGBUS_DATABASE_URL` run (run-and-report, never a CI gate). Groundwork for the elastic-pools research in #323; a `pgbus doctor` mode/pool-mismatch hint is deferred to a follow-up. Refs #323. - **Off-thread durable-stream fanout writer (`config.streams_writer_threads`, `config.streams_writer_buffer_limit`) — the full head-of-line cure the #315 deadline only bounded.** The #315 deadline caps each slow-client fanout stall at `streams_fanout_write_deadline_ms` (250ms), but the single dispatcher thread still writes to every client serially, so K slow clients still cost ~`K × 250ms`. Setting `streams_writer_threads > 0` moves the **durable** fanout socket write off the dispatcher into N writer threads (each connection pinned to one worker by `id.hash % N`, preserving per-connection frame order and its per-io mutex). The dispatcher hands off the write and moves on; the pump reports back the highest msg_id it actually committed via an ack queue, and the dispatcher — still the **sole owner** of the read cursor — advances it only on that ack, so a blocked or partial write never advances the cursor past a frame that didn't reach the socket. A failed write posts a `DisconnectMessage` so the dispatcher scrubs the connection deterministically on its own thread (even on an otherwise-quiet stream). Measured (`streams_writer_offload_bench.rb`, dispatcher `handle_durable_wake` wall time, 50 fast + K slow clients @ 50ms): inline scales linearly (K=20 → **1071 ms**) while offload stays flat (K=20 → **0.09 ms**) — fast clients no longer wait behind slow ones. This is a **system-level latency win, not a per-write speedup** (total bytes written are unchanged); the dispatcher does slightly more per-wake bookkeeping under offload, which is why it is **opt-in and default-off** (`streams_writer_threads = 0` = byte-for-byte the pre-#321 inline behavior). **Ephemeral frames always stay inline** regardless of this setting — they have no archive to replay, so they must not risk an async drop (the pump raises if one is ever routed to it; issue #321 B1). `streams_writer_buffer_limit` (default `0` = unbounded) caps a connection's outbound buffer, dropping its oldest durable frame on overflow — safe because durable frames are re-read from the archive on reconnect; it's an OOM guard for a pathologically slow-but-alive client, not a delivery guarantee. Both knobs are dispatcher-internal tuning (not part of the 1.0 public-config surface). Refs #321 (follow-up to #315). diff --git a/Rakefile b/Rakefile index 1dd9050..5b47af6 100644 --- a/Rakefile +++ b/Rakefile @@ -24,7 +24,7 @@ namespace :bench do # Benches that need a real PostgreSQL/PGMQ (or boot Puma) — excluded from the # no-DB unit suite that bench:all runs in CI. db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench - execution_modes_bench pool_swap_bench pool_autoscale_bench].freeze + execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench].freeze # The unit suite is every *_bench.rb that doesn't need a database, derived # from the directory so a new unit bench is picked up automatically (kept in # sync with bench:one, which globs the same files). @@ -79,6 +79,11 @@ namespace :bench do ruby "benchmarks/pool_autoscale_bench.rb" end + desc "Run job-burst gate benchmark (#323 phase 3: is the job pool or DB pool the burst limiter?; requires PGBUS_DATABASE_URL)" + task :job_burst do + ruby "benchmarks/job_burst_bench.rb" + end + desc "Run a single benchmark: rake bench:one[client_bench]" task :one, [:name] do |_t, args| name = args[:name] or abort "Usage: rake bench:one[serialization_bench|client_bench|...]" diff --git a/benchmarks/job_burst_bench.rb b/benchmarks/job_burst_bench.rb new file mode 100644 index 0000000..29d7fa0 --- /dev/null +++ b/benchmarks/job_burst_bench.rb @@ -0,0 +1,152 @@ +# frozen_string_literal: true + +# Job-burst gate benchmark (issue #323 phase 3 — the measure-first gate). +# +# QUESTION IT ANSWERS: under a job burst, is the JOB EXECUTION POOL (thread +# count) the throughput limiter, or is it the DB CONNECTION POOL (pgmq +# checkout)? Issue #323 phase 3 mandates this measurement before building +# elastic job pools — if raising the static `threads:` (with a matching +# `pool_size`) already recovers burst throughput, then "static headroom is the +# answer" and no elastic machinery is warranted. +# +# HOW: for a fixed burst of jobs, sweep execution-pool concurrency (thread +# count) upward under two connection regimes, using the shared +# ExecutionModeHarness#run_cell (same rig as #324): +# +# (1) MATCHED — pool_size == concurrency (the "raise threads AND pool_size" +# static-headroom answer). If throughput keeps climbing with +# concurrency here, more threads help → the job pool is (part +# of) the limiter, and static headroom already fixes it. +# (2) STARVED — pool_size fixed small while concurrency climbs (the +# "elastic threads, fixed connection pool" case — exactly what +# elastic job threads WITHOUT elastic connections would give). +# If throughput plateaus at ~pool_size regardless of thread +# count, then threads past the connection ceiling buy nothing +# → elastic job threads alone are a no-op. +# +# Reading the two curves together tells you which pool to make elastic (if any). +# Expectation from #324 (job holds a pgmq connection only for the read/archive +# round-trip): the STARVED curve plateaus at the connection ceiling, so raising +# threads past `resolved_pool_size` doesn't help — static headroom (raise BOTH) +# is the fix. This bench confirms or refutes that on your hardware. +# +# Requires PGBUS_DATABASE_URL. Run-and-report, never a CI gate. +# PGBUS_DATABASE_URL=postgres://user@host/db bundle exec rake bench:job_burst + +require "logger" +require "pg" +require "pgbus" +require_relative "support/execution_mode_harness" + +DATABASE_URL = ENV.fetch("PGBUS_DATABASE_URL") do + warn "PGBUS_DATABASE_URL not set. This benchmark requires a real PostgreSQL database." + warn "Example: PGBUS_DATABASE_URL=postgres://user@localhost/pgbus_test bundle exec rake bench:job_burst" + exit 1 +end + +MIN_FREE_BACKENDS = 30 +JOB_COUNT = 400 # the burst size (same for every cell → fair) +CONCURRENCY_SWEEP = [2, 4, 8, 16].freeze +STARVED_POOL_SIZE = 4 # fixed small connection ceiling for regime (2) +# DB-bound profile: each job spends most of its time INSIDE a pooled checkout +# (SELECT pg_sleep), so the connection pool is the contended resource — the +# regime where "more threads vs more connections" actually diverges. +IO_PROFILE = ExecutionModeHarness::IoProfile.db_bound + +def build_client(pool_size) + config = Pgbus::Configuration.new.tap do |c| + c.database_url = DATABASE_URL + c.queue_prefix = "pgbus_jobburst" + c.default_queue = "default" + c.logger = Logger.new(IO::NULL) + c.pool_size = pool_size + c.pool_timeout = 5 + c.stats_enabled = false + end + Pgbus::Client.new(config, schema_ensured: true) +end + +def free_backends + conn = PG.connect(DATABASE_URL) + total = conn.exec("SHOW max_connections").first["max_connections"].to_i + used = conn.exec("SELECT count(*) AS n FROM pg_stat_activity").first["n"].to_i + total - used +ensure + conn&.close +end + +def run_cell(pool_size:, concurrency:) + client = build_client(pool_size) + ExecutionModeHarness.run_cell( + mode: :threads, pool_size: pool_size, concurrency: concurrency, + io_profile: IO_PROFILE, job_count: JOB_COUNT, client: client + ) +ensure + client&.close +end + +puts "=" * 76 +puts "Job-burst gate benchmark (issue #323 phase 3)" +puts "Database: #{DATABASE_URL.sub(%r{//[^@]+@}, "//***@")}" +puts "Burst: #{JOB_COUNT} db-bound jobs (db=#{IO_PROFILE.db_seconds}s in-checkout each)" +puts "=" * 76 + +free = free_backends +if free < MIN_FREE_BACKENDS + warn "Only #{free} Postgres backends free (need >= #{MIN_FREE_BACKENDS}); free some and retry." + exit 1 +end + +matched = CONCURRENCY_SWEEP.map { |c| run_cell(pool_size: c, concurrency: c) } +starved = CONCURRENCY_SWEEP.map { |c| run_cell(pool_size: STARVED_POOL_SIZE, concurrency: c) } + +def print_table(title, results) + puts + puts title + headers = ExecutionModeHarness::Result.headers + rows = results.map(&:to_row) + widths = headers.map.with_index { |h, i| [h.length, *rows.map { |r| r[i].to_s.length }].max } + fmt = ->(row) { row.each_with_index.map { |v, i| v.to_s.ljust(widths[i]) }.join(" ") } + puts fmt.call(headers) + rows.each { |r| puts fmt.call(r) } +end + +print_table("(1) MATCHED — pool_size == concurrency (the static-headroom answer)", matched) +print_table("(2) STARVED — pool_size fixed at #{STARVED_POOL_SIZE} (elastic threads, fixed connections)", starved) + +# ─── Verdict ─── +# The decisive signal is peak_busy, NOT raw throughput: peak_busy = the max live +# checkouts, which cannot exceed pool_size. If, in the STARVED regime, peak_busy +# caps at pool_size while concurrency climbs past it, then the extra threads are +# just waiting on connection checkout — they cannot do more concurrent work. A +# db-bound job still has a tiny non-checkout slice, so a couple extra threads can +# overlap it for a small one-time throughput bump; that is NOT the threads being +# the limiter, so we key the verdict off peak_busy capping, not a throughput ratio. +matched_thr = matched.map(&:throughput) +starved_thr = starved.map(&:throughput) +matched_gain = matched_thr.last / matched_thr.first +starved_peak_busy = starved.map(&:peak_busy).max +# Did adding threads beyond pool_size raise concurrent work (peak_busy)? +capped_at_pool = starved_peak_busy <= STARVED_POOL_SIZE + +puts +puts "Reading the curves:" +puts format(" MATCHED throughput %.0f → %.0f jobs/s across concurrency %s (%.1f× gain, peak_busy tracks conc)", + matched_thr.first, matched_thr.last, CONCURRENCY_SWEEP.inspect, matched_gain) +puts format(" STARVED throughput %.0f → %.0f jobs/s; peak_busy capped at %d (pool_size=%d)", + starved_thr.first, starved_thr.last, starved_peak_busy, STARVED_POOL_SIZE) +puts +if capped_at_pool + puts "VERDICT: in the STARVED regime peak_busy never exceeds pool_size (#{STARVED_POOL_SIZE}) no matter" + puts "how many threads run — extra threads just WAIT on connection checkout (see p99 latency" + puts "climb). The DB connection pool, not the thread pool, is the hard burst ceiling. Elastic" + puts "job THREADS alone (fixed connection pool) cannot push past it. The fix is STATIC HEADROOM:" + puts "raise BOTH `threads:` and `pool_size:` — the MATCHED curve scales linearly when you do." + puts "This is issue #323's sanctioned outcome. Do NOT build elastic job pools." +else + puts "VERDICT: STARVED peak_busy exceeded pool_size — extra threads did more concurrent work" + puts "despite the fixed connection pool. The thread pool is an independent limiter; elastic job" + puts "threads (issue #323 phase 3b, Design A) may be worth building. Unexpected given #324 —" + puts "re-check the job's connection-holding profile." +end +puts "Done." diff --git a/docs/app/views/docs/pages/performance_tuning.rb b/docs/app/views/docs/pages/performance_tuning.rb index 9ddcbc7..c821f92 100644 --- a/docs/app/views/docs/pages/performance_tuning.rb +++ b/docs/app/views/docs/pages/performance_tuning.rb @@ -12,12 +12,53 @@ def lead = "Tune autovacuum for the high-churn queue tables, size archive retent def content autovacuum archive + job_burst_tuning streams_pool_autoscaling health_metrics end private + def job_burst_tuning + DocsUI::Section("Job bursts: raise threads and the pool together", + description: "Under a job spike, the DB connection pool is the ceiling — not the thread count.") do + md <<~'MD' + When a queue floods, the instinct is to add worker threads. But a job + holds a database connection only for the brief `read_batch` + `archive` + round-trip — not for the job body — so a worker's throughput is capped by + its **connection pool**, not its thread count. Adding threads past the + pool size just makes them queue on connection checkout: latency climbs, + throughput doesn't. + MD + md <<~'MD' + So size the two **together**. Raising `threads` alone plateaus at the pool + size; raising both scales throughput roughly linearly (measured 8× from + 2→16 when the pool matches). The connection pool auto-tunes from the thread + count by default, so in practice you raise `threads` and let `pool_size` + follow — but if you pin `pool_size`, keep it ≥ `threads`. + MD + DocsUI::Code(<<~RUBY, filename: "config/initializers/pgbus.rb") + Pgbus.configure do |config| + config.worker "default", threads: 16 # more concurrency… + config.pool_size = 20 # …needs the connections to back it + end + RUBY + DocsUI::Callout(:note) do + plain "This is the " + strong { "static headroom" } + plain " answer, and it's usually the right one: idle pool slots are lazy — " + plain "they cost nothing until a burst uses them. pgbus deliberately does " + plain "not autoscale the job pool, because elastic threads on a fixed " + plain "connection pool can't push past the connection ceiling (measured " + plain "with " + code { "rake bench:job_burst" } + plain "). Watch " + code { "pgbus_worker_pool_utilization" } + plain " — sustained near 1 means raise both." + end + end + end + def streams_pool_autoscaling DocsUI::Section("Streams pool autoscaling", description: "Let the SSE streams pool grow into spare connections under a burst, and shrink back when it's over.") do diff --git a/docs/performance.md b/docs/performance.md index f2a38a6..6bb007c 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -27,6 +27,7 @@ to guess. | Streamer replay read | every durable-stream wake + SSE connect (`Client#read_after`) | `streams_read_pool_bench.rb` | | Streams-pool hot-swap | elastic resize under load (issue #323) | `pool_swap_bench.rb` | | Streams-pool autoscaler | per-tick decision cost + grow-under-load (issue #323) | `pool_autoscale_bench.rb` | +| Job-burst limiter gate | is the job pool or the DB pool the burst ceiling? (issue #323 phase 3) | `job_burst_bench.rb` | ## Measuring