perf(jobs): job-burst gate bench + static-headroom guidance; don't autoscale the job pool (#323 phase 3)

CHANGELOG.md

@@ -7,6 +7,7 @@
 
 ### Added
 
+- **Job-burst tuning guidance + gate benchmark (`rake bench:job_burst`), and a decision NOT to autoscale the job pool (issue #323 phase 3).** Following the issue's measure-first mandate, `benchmarks/job_burst_bench.rb` sweeps worker thread count under a job burst in two connection regimes (pool matched to threads vs. pool fixed small). Measured finding: a job holds a pgmq connection only for the `read_batch` + `archive` round-trip (not the job body), so **the DB connection pool is the hard burst ceiling, not the thread count** — in the fixed-pool regime `peak_busy` never exceeds `pool_size` no matter how many threads run (extra threads just queue on checkout, latency climbs, throughput plateaus), while matching the pool to the thread count scales throughput ~linearly (≈8× from 2→16). The conclusion — issue #323's explicitly sanctioned outcome — is that **elastic job threads on a fixed connection pool cannot help**, so pgbus does *not* autoscale the job execution pool; the fix for job bursts is **static headroom**: raise `threads` (and let `pool_size` follow, or keep it ≥ `threads`). Documented in the Performance & tuning docs. The streams-pool autoscaling below stays the elastic story where it *does* pay off (the streams pool's connection cost genuinely scales with its size). Refs #323.
 - **Self-tuning elastic streams-pool autoscaling (`config.streams_pool_autoscale`, opt-in, default off).** The dedicated streams pool (durable-broadcast publish + dispatcher replay reads) is a fixed `streams_pool_size` (default 5); under a genuine burst of SSE clients it saturates, and a saturated pool *serialises* replay reads (it doesn't error — the checkout waits; see #324), slowing fan-out. Setting `config.streams_pool_autoscale = true` runs a **periodic maintenance check** (default every 5 minutes, `config.streams_pool_autoscale_interval`) that **grows the streams pool into a fair share of live Postgres connection headroom** while it's saturated and **shrinks it back to `streams_pool_size`** when the burst passes — with no connection-count target to tune: every threshold derives from live `max_connections`. The check adds **no extra connection and no extra thread** — it's a lightweight `pg_stat_activity` query that runs on the streamer's *existing idle LISTEN connection* (a pghero-`capture_query_stats`-style periodic task, run on the listener thread in its health-check window). It counts peer stream processes by a per-process `application_name` tag (`config.streams_application_name`, default `"pgbus_streams"`), and each process grows only into its own fair share of free connections (`free / peers`, claiming ¼, bounded by a per-grow step and half the remaining headroom) so N forked processes provably can't collectively exhaust the database — even in a synchronized cold-boot herd. If the database runs *critically* low on free connections, every process **emergency-shrinks to baseline immediately**, overriding the saturation signal (protecting the DB wins). `config.streams_pool_max` is an **optional** hard per-process ceiling (`nil` = the dynamic fair share is the cap). No per-sample hysteresis or cooldown is needed — the slow cadence is itself the debounce: one grow (or shrink) step per check, and a sustained burst converges over a few checks. Built on the #325 hot-swap primitive; the decision object is a pure function (no thread, no connection), DB-free unit-tested (scripted headroom, incl. convergence-across-checks and emergency-shrink), with a DB-gated integration test proving real grow/shrink/emergency and a `rake bench:pool_autoscale` benchmark (check cost ~0.3 ms; grow-under-saturation `3→7→10`). **Two prerequisite fixes ship with it:** streams-pool connections are now tagged with a per-process `application_name`, and the streams pool is now built from `streams_connection_options` (honoring `streams_database_url`/host/port) rather than the job-database options — the latter a latent wrong-DB bug in the existing streams pool that only surfaced with a separate streams database. Default-off = byte-identical (the Listener runs no maintenance). A pure-*publisher* worker process (no streamer, only `send_stream_message`) autoscales too: each publish opportunistically triggers the same check, throttled to at most once per `streams_pool_autoscale_interval` across all publisher threads (a lock-free compare-and-set claims the window). The publish thread does only that CAS and then hands the work to a background single-worker executor — it never runs the `pg_stat_activity` query or the resize inline (off the hot path), and a quiet process spawns no thread. The query runs through the *job* pool (so it never competes with the streams pool it grows), and the publisher decision is **grow-only**: it grows under its own publish pressure and emergency-shrinks on DB exhaustion, but leaves normal idle shrink to the streamer/consumer that sees the sustained-idle picture. Fail-soft throughout, so a broadcast never fails on a telemetry query. For steady load, a larger static `streams_pool_size` remains the simpler choice — autoscaling earns its keep under genuinely bursty streams traffic. `PoolAutoscaler` lives in `Pgbus::Streams::` (it is a streams-pool controller, not web-specific). Refs #323.
 - **Execution-mode connection benchmark + DB-pool sizing guidance (`rake bench:execution_modes`).** A fair, reusable harness (`benchmarks/support/execution_mode_harness.rb` + `benchmarks/execution_modes_bench.rb`) that runs the *same* offered load through the threads and async execution pools and measures how many real Postgres connections each actually holds (`peak_busy = pool size − available`, sampled through the real pool checkout), so operators can size the DB pool per `execution_mode`. It settles the "async saves connections" question with data: a job holds a pgmq connection only for the `read_batch` + `archive` round-trip (not the job body), so **async's connection-density win is real only for I/O-light work** (a few connections serve many fibers) — for **DB-bound work async is connection-bound like threads**, and a too-small pool *serialises* (throughput collapse, ~3.4× in the measured cell) rather than erroring, because `pool_timeout` (5 s) dwarfs a typical checkout. New `docs/performance.md` section "Execution mode and DB pool sizing" carries the measured numbers and a sizing table. The harness's fairness/struct/measurement logic is unit-tested with fakes (no DB, runs in CI); the numbers come from a manual `PGBUS_DATABASE_URL` run (run-and-report, never a CI gate). Groundwork for the elastic-pools research in #323; a `pgbus doctor` mode/pool-mismatch hint is deferred to a follow-up. Refs #323.
 - **Off-thread durable-stream fanout writer (`config.streams_writer_threads`, `config.streams_writer_buffer_limit`) — the full head-of-line cure the #315 deadline only bounded.** The #315 deadline caps each slow-client fanout stall at `streams_fanout_write_deadline_ms` (250ms), but the single dispatcher thread still writes to every client serially, so K slow clients still cost ~`K × 250ms`. Setting `streams_writer_threads > 0` moves the **durable** fanout socket write off the dispatcher into N writer threads (each connection pinned to one worker by `id.hash % N`, preserving per-connection frame order and its per-io mutex). The dispatcher hands off the write and moves on; the pump reports back the highest msg_id it actually committed via an ack queue, and the dispatcher — still the **sole owner** of the read cursor — advances it only on that ack, so a blocked or partial write never advances the cursor past a frame that didn't reach the socket. A failed write posts a `DisconnectMessage` so the dispatcher scrubs the connection deterministically on its own thread (even on an otherwise-quiet stream). Measured (`streams_writer_offload_bench.rb`, dispatcher `handle_durable_wake` wall time, 50 fast + K slow clients @ 50ms): inline scales linearly (K=20 → **1071 ms**) while offload stays flat (K=20 → **0.09 ms**) — fast clients no longer wait behind slow ones. This is a **system-level latency win, not a per-write speedup** (total bytes written are unchanged); the dispatcher does slightly more per-wake bookkeeping under offload, which is why it is **opt-in and default-off** (`streams_writer_threads = 0` = byte-for-byte the pre-#321 inline behavior). **Ephemeral frames always stay inline** regardless of this setting — they have no archive to replay, so they must not risk an async drop (the pump raises if one is ever routed to it; issue #321 B1). `streams_writer_buffer_limit` (default `0` = unbounded) caps a connection's outbound buffer, dropping its oldest durable frame on overflow — safe because durable frames are re-read from the archive on reconnect; it's an OOM guard for a pathologically slow-but-alive client, not a delivery guarantee. Both knobs are dispatcher-internal tuning (not part of the 1.0 public-config surface). Refs #321 (follow-up to #315).

Rakefile

@@ -24,7 +24,7 @@ namespace :bench do
   # Benches that need a real PostgreSQL/PGMQ (or boot Puma) — excluded from the
   # no-DB unit suite that bench:all runs in CI.
   db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
-                  execution_modes_bench pool_swap_bench pool_autoscale_bench].freeze
+                  execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench].freeze
   # The unit suite is every *_bench.rb that doesn't need a database, derived
   # from the directory so a new unit bench is picked up automatically (kept in
   # sync with bench:one, which globs the same files).
@@ -79,6 +79,11 @@ namespace :bench do
     ruby "benchmarks/pool_autoscale_bench.rb"
   end
 
+  desc "Run job-burst gate benchmark (#323 phase 3: is the job pool or DB pool the burst limiter?; requires PGBUS_DATABASE_URL)"
+  task :job_burst do
+    ruby "benchmarks/job_burst_bench.rb"
+  end
+
   desc "Run a single benchmark: rake bench:one[client_bench]"
   task :one, [:name] do |_t, args|
     name = args[:name] or abort "Usage: rake bench:one[serialization_bench|client_bench|...]"

benchmarks/job_burst_bench.rb

@@ -0,0 +1,152 @@
+# frozen_string_literal: true
+
+# Job-burst gate benchmark (issue #323 phase 3 — the measure-first gate).
+#
+# QUESTION IT ANSWERS: under a job burst, is the JOB EXECUTION POOL (thread
+# count) the throughput limiter, or is it the DB CONNECTION POOL (pgmq
+# checkout)? Issue #323 phase 3 mandates this measurement before building
+# elastic job pools — if raising the static `threads:` (with a matching
+# `pool_size`) already recovers burst throughput, then "static headroom is the
+# answer" and no elastic machinery is warranted.
+#
+# HOW: for a fixed burst of jobs, sweep execution-pool concurrency (thread
+# count) upward under two connection regimes, using the shared
+# ExecutionModeHarness#run_cell (same rig as #324):
+#
+#   (1) MATCHED   — pool_size == concurrency (the "raise threads AND pool_size"
+#                   static-headroom answer). If throughput keeps climbing with
+#                   concurrency here, more threads help → the job pool is (part
+#                   of) the limiter, and static headroom already fixes it.
+#   (2) STARVED   — pool_size fixed small while concurrency climbs (the
+#                   "elastic threads, fixed connection pool" case — exactly what
+#                   elastic job threads WITHOUT elastic connections would give).
+#                   If throughput plateaus at ~pool_size regardless of thread
+#                   count, then threads past the connection ceiling buy nothing
+#                   → elastic job threads alone are a no-op.
+#
+# Reading the two curves together tells you which pool to make elastic (if any).
+# Expectation from #324 (job holds a pgmq connection only for the read/archive
+# round-trip): the STARVED curve plateaus at the connection ceiling, so raising
+# threads past `resolved_pool_size` doesn't help — static headroom (raise BOTH)
+# is the fix. This bench confirms or refutes that on your hardware.
+#
+# Requires PGBUS_DATABASE_URL. Run-and-report, never a CI gate.
+#   PGBUS_DATABASE_URL=postgres://user@host/db bundle exec rake bench:job_burst
+
+require "logger"
+require "pg"
+require "pgbus"
+require_relative "support/execution_mode_harness"
+
+DATABASE_URL = ENV.fetch("PGBUS_DATABASE_URL") do
+  warn "PGBUS_DATABASE_URL not set. This benchmark requires a real PostgreSQL database."
+  warn "Example: PGBUS_DATABASE_URL=postgres://user@localhost/pgbus_test bundle exec rake bench:job_burst"
+  exit 1
+end
+
+MIN_FREE_BACKENDS = 30
+JOB_COUNT = 400                       # the burst size (same for every cell → fair)
+CONCURRENCY_SWEEP = [2, 4, 8, 16].freeze
+STARVED_POOL_SIZE = 4                 # fixed small connection ceiling for regime (2)
+# DB-bound profile: each job spends most of its time INSIDE a pooled checkout
+# (SELECT pg_sleep), so the connection pool is the contended resource — the
+# regime where "more threads vs more connections" actually diverges.
+IO_PROFILE = ExecutionModeHarness::IoProfile.db_bound
+
+def build_client(pool_size)
+  config = Pgbus::Configuration.new.tap do |c|
+    c.database_url = DATABASE_URL
+    c.queue_prefix = "pgbus_jobburst"
+    c.default_queue = "default"
+    c.logger = Logger.new(IO::NULL)
+    c.pool_size = pool_size
+    c.pool_timeout = 5
+    c.stats_enabled = false
+  end
+  Pgbus::Client.new(config, schema_ensured: true)
+end
+
+def free_backends
+  conn = PG.connect(DATABASE_URL)
+  total = conn.exec("SHOW max_connections").first["max_connections"].to_i
+  used  = conn.exec("SELECT count(*) AS n FROM pg_stat_activity").first["n"].to_i
+  total - used
+ensure
+  conn&.close
+end
+
+def run_cell(pool_size:, concurrency:)
+  client = build_client(pool_size)
+  ExecutionModeHarness.run_cell(
+    mode: :threads, pool_size: pool_size, concurrency: concurrency,
+    io_profile: IO_PROFILE, job_count: JOB_COUNT, client: client
+  )
+ensure
+  client&.close
+end
+
+puts "=" * 76
+puts "Job-burst gate benchmark (issue #323 phase 3)"
+puts "Database: #{DATABASE_URL.sub(%r{//[^@]+@}, "//***@")}"
+puts "Burst: #{JOB_COUNT} db-bound jobs (db=#{IO_PROFILE.db_seconds}s in-checkout each)"
+puts "=" * 76
+
+free = free_backends
+if free < MIN_FREE_BACKENDS
+  warn "Only #{free} Postgres backends free (need >= #{MIN_FREE_BACKENDS}); free some and retry."
+  exit 1
+end
+
+matched = CONCURRENCY_SWEEP.map { |c| run_cell(pool_size: c, concurrency: c) }
+starved = CONCURRENCY_SWEEP.map { |c| run_cell(pool_size: STARVED_POOL_SIZE, concurrency: c) }
+
+def print_table(title, results)
+  puts
+  puts title
+  headers = ExecutionModeHarness::Result.headers
+  rows = results.map(&:to_row)
+  widths = headers.map.with_index { |h, i| [h.length, *rows.map { |r| r[i].to_s.length }].max }
+  fmt = ->(row) { row.each_with_index.map { |v, i| v.to_s.ljust(widths[i]) }.join("  ") }
+  puts fmt.call(headers)
+  rows.each { |r| puts fmt.call(r) }
+end
+
+print_table("(1) MATCHED — pool_size == concurrency  (the static-headroom answer)", matched)
+print_table("(2) STARVED — pool_size fixed at #{STARVED_POOL_SIZE}  (elastic threads, fixed connections)", starved)
+
+# ─── Verdict ───
+# The decisive signal is peak_busy, NOT raw throughput: peak_busy = the max live
+# checkouts, which cannot exceed pool_size. If, in the STARVED regime, peak_busy
+# caps at pool_size while concurrency climbs past it, then the extra threads are
+# just waiting on connection checkout — they cannot do more concurrent work. A
+# db-bound job still has a tiny non-checkout slice, so a couple extra threads can
+# overlap it for a small one-time throughput bump; that is NOT the threads being
+# the limiter, so we key the verdict off peak_busy capping, not a throughput ratio.
+matched_thr = matched.map(&:throughput)
+starved_thr = starved.map(&:throughput)
+matched_gain = matched_thr.last / matched_thr.first
+starved_peak_busy = starved.map(&:peak_busy).max
+# Did adding threads beyond pool_size raise concurrent work (peak_busy)?
+capped_at_pool = starved_peak_busy <= STARVED_POOL_SIZE
+
+puts
+puts "Reading the curves:"
+puts format("  MATCHED throughput %.0f → %.0f jobs/s across concurrency %s (%.1f× gain, peak_busy tracks conc)",
+            matched_thr.first, matched_thr.last, CONCURRENCY_SWEEP.inspect, matched_gain)
+puts format("  STARVED throughput %.0f → %.0f jobs/s; peak_busy capped at %d (pool_size=%d)",
+            starved_thr.first, starved_thr.last, starved_peak_busy, STARVED_POOL_SIZE)
+puts
+if capped_at_pool
+  puts "VERDICT: in the STARVED regime peak_busy never exceeds pool_size (#{STARVED_POOL_SIZE}) no matter"
+  puts "how many threads run — extra threads just WAIT on connection checkout (see p99 latency"
+  puts "climb). The DB connection pool, not the thread pool, is the hard burst ceiling. Elastic"
+  puts "job THREADS alone (fixed connection pool) cannot push past it. The fix is STATIC HEADROOM:"
+  puts "raise BOTH `threads:` and `pool_size:` — the MATCHED curve scales linearly when you do."
+  puts "This is issue #323's sanctioned outcome. Do NOT build elastic job pools."
+else
+  puts "VERDICT: STARVED peak_busy exceeded pool_size — extra threads did more concurrent work"
+  puts "despite the fixed connection pool. The thread pool is an independent limiter; elastic job"
+  puts "threads (issue #323 phase 3b, Design A) may be worth building. Unexpected given #324 —"
+  puts "re-check the job's connection-holding profile."
+end
+puts "Done."

docs/app/views/docs/pages/performance_tuning.rb

@@ -12,12 +12,53 @@ def lead = "Tune autovacuum for the high-churn queue tables, size archive retent
   def content
     autovacuum
     archive
+    job_burst_tuning
     streams_pool_autoscaling
     health_metrics
   end
 
   private
 
+  def job_burst_tuning
+    DocsUI::Section("Job bursts: raise threads and the pool together",
+                    description: "Under a job spike, the DB connection pool is the ceiling — not the thread count.") do
+      md <<~'MD'
+        When a queue floods, the instinct is to add worker threads. But a job
+        holds a database connection only for the brief `read_batch` + `archive`
+        round-trip — not for the job body — so a worker's throughput is capped by
+        its **connection pool**, not its thread count. Adding threads past the
+        pool size just makes them queue on connection checkout: latency climbs,
+        throughput doesn't.
+      MD
+      md <<~'MD'
+        So size the two **together**. Raising `threads` alone plateaus at the pool
+        size; raising both scales throughput roughly linearly (measured 8× from
+        2→16 when the pool matches). The connection pool auto-tunes from the thread
+        count by default, so in practice you raise `threads` and let `pool_size`
+        follow — but if you pin `pool_size`, keep it ≥ `threads`.
+      MD
+      DocsUI::Code(<<~RUBY, filename: "config/initializers/pgbus.rb")
+        Pgbus.configure do |config|
+          config.worker "default", threads: 16   # more concurrency…
+          config.pool_size = 20                  # …needs the connections to back it
+        end
+      RUBY
+      DocsUI::Callout(:note) do
+        plain "This is the "
+        strong { "static headroom" }
+        plain " answer, and it's usually the right one: idle pool slots are lazy — "
+        plain "they cost nothing until a burst uses them. pgbus deliberately does "
+        plain "not autoscale the job pool, because elastic threads on a fixed "
+        plain "connection pool can't push past the connection ceiling (measured "
+        plain "with "
+        code { "rake bench:job_burst" }
+        plain "). Watch "
+        code { "pgbus_worker_pool_utilization" }
+        plain " — sustained near 1 means raise both."
+      end
+    end
+  end
+
   def streams_pool_autoscaling
     DocsUI::Section("Streams pool autoscaling",
                     description: "Let the SSE streams pool grow into spare connections under a burst, and shrink back when it's over.") do