build: cap the host at two compiling builds over six rustc tokens

.claude/skills/cpu-reduction/SKILL.md

@@ -37,7 +37,7 @@ df -h ~/.cache                            # disk — worktree target/ dirs live
 ps -Ao pid,ppid,pgid,%cpu,%mem,rss,etime,state,comm -r | head -40
 ```
 
-Load well above `hw.ncpu` with nothing building → §3. A build stuck on a lock, or `~/.cache` near full → §2.
+Load well above `hw.ncpu` with nothing building → §3. A build stuck on a lock, or `~/.cache` near full → §2. Start with `make build-status`: it prints which builds hold the machine-wide compile slots, who is queued behind them and for how long, token occupancy, and any build running outside the cap — a build sitting at `Compiling` with no CPU is queued behind the governor, not hung.
 
 ## 2. Old Rust builds — locks, orphans, disk
 

.claude/skills/dev-loop/SKILL.md

@@ -62,8 +62,7 @@ There is **no `nub build` command**.
 
 **Build politeness — the maintainer works on this machine.**
 
-- **Job cap (already set, machine-wide):** `~/.cargo/config.toml` pins `[build] jobs = 6` of 8 perf cores. CI is unaffected. Leave it in place.
-- **Background QoS — wrap every agent build:** `taskpolicy -b cargo build -p nub-cli --profile fast` (macOS background QoS → E-cores, yields to interactive) or `nice -n 10 cargo build …`. For a build already hammering the host: `renice 20 -p <pid>` + `taskpolicy -b -p <pid>` on the running `cargo`/`rustc` tree.
+- **The machine-wide governor does the clamping, not you.** `~/.cargo/config.toml` binds `scripts/rustc-qos.sh` as the rustc wrapper (`make qos-global`; `install-dev` re-runs it), which runs every rustc at utility QoS, lets at most two builds compile at once (the rest queue first-come-first-served), and caps the host at six rustc. So a build that sits at `Compiling` with no CPU is queued, not hung: `make build-status` shows who holds the slots and who is waiting. Do not wrap builds in `taskpolicy`/`nice` yourself, and never set `NUB_BUILD_FG=1` or blank `RUSTC_WRAPPER` from an agent — the `rust-build-hygiene` skill has the rules. `~/.cargo/config.toml` also pins `[build] jobs = 6`; CI is unaffected. Leave both in place.
 
 **Why `fast`, never `release`, for iteration** (measured, macOS arm64):
 

.claude/skills/rust-build-hygiene/SKILL.md

@@ -48,9 +48,9 @@ Never fake-wait on a build with a detached shell + a `sleep`/poll loop.
 
 **A per-build cap cannot bound N builds.** Every build can be individually blameless — `--profile fast`, QoS-clamped, `jobs = 6` — and the machine still dies, because the caps multiply instead of adding. Measured 2026-08-19: 13 concurrent agent builds, every one of them already on `--profile fast`, produced a 78-way oversubscription of 10 cores — load 464, 0% idle, 36% sys, and a `reqwest` compile that normally takes ~30s taking 28 minutes. Nothing was misconfigured. There was simply no cap on the SUM.
 
-- **`make qos-global` installs the global governor and is what actually bounds the fleet.** It registers `scripts/rustc-qos.sh` as the machine-wide rustc wrapper, where it does two jobs: clamp QoS, and hold one of `ncpu` tokens for the life of each rustc. Every build on the host shares that one pool, so one build gets the whole machine and thirteen share it. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script.
-- **Never blank `RUSTC_WRAPPER`.** That is cargo's documented "no wrapper", and it opts the build out of the global cap. `scripts/rust-build.sh` used to do exactly this, which is why 10 of those 13 builds were ungoverned. `NUB_BUILD_FG=1` is the supported opt-out for a latency-sensitive foreground build.
-- **`make build-status` answers "why is this machine saturated?"** It prints the sum no single session can see: load, live builds, semaphore occupancy, and which builds are outside the cap. Run it before concluding your own build is slow — it usually is not your build.
+- **`make qos-global` installs the global governor and is what actually bounds the fleet.** It registers `scripts/rustc-qos.sh` as the machine-wide rustc wrapper, where it does three jobs: clamp QoS; **let at most TWO builds compile at a time** (`NUB_BUILD_SLOTS`, default 2 — every other build's first rustc waits in a first-come-first-served queue until a holder's cargo exits, dies, or goes idle for `NUB_BUILD_IDLE`=120s, so a `cargo test` running its tests or a cargo blocked on a target lock does not hold the machine); and across the compiling builds, hold one of `NUB_RUSTC_LIMIT` (default 6) tokens for the life of each rustc. Two builds over six tokens bounds the memory peak to ~6 big-crate compiles (~12 GiB) and keeps a second build's worth of cores busy; strict one-at-a-time was the first cut (2026-08-28) and was measured idling nine cores behind one starved compile while eight builds queued 25 minutes. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script. A build that queues is not stuck — after 20s it prints one `rustc-qos: this build is queued …` line on its cargo's stderr, and `make build-status` shows the holders and the queue. `rust-analyzer` is exempt so the editor never waits behind agent builds.
+- **Never blank `RUSTC_WRAPPER`, and never set `NUB_BUILD_FG=1` from an agent.** Blanking is cargo's documented "no wrapper", and it opts the build out of the global cap; `scripts/rust-build.sh` used to do exactly this, which is why 10 of those 13 builds were ungoverned. `NUB_BUILD_FG=1` opts a build out of the QoS clamp and the build-slot queue (a bare `cargo` still takes rustc tokens; through `rust-build.sh` it blanks both wrapper keys, so out of the tokens too) — it exists for a HUMAN at a terminal whose build must not wait behind the fleet, and an agent that sets it recreates the 2026-08-19 incident. `NUB_BUILD_SLOTS=0` disables only the slot layer and `NUB_BUILD_SLOTS=1` restores strict one-at-a-time — both are PER-PROCESS environment knobs, read by the wrapper of the cargo that inherits them, not host-wide settings. The host-wide switch is `make build-slots-off` / `build-slots-on` (a file every wrapper checks each second, so it also releases builds already queued); it is for an emergency, and it leaves the QoS clamp and the tokens in place.
+- **`make build-status` answers "why is this machine saturated?" and "why is my build not starting?"** It prints the sum no single session can see: load, which builds hold the compile slots and who is queued behind them (each tagged with its worktree), token occupancy, a STALE WRAPPER line when an older checkout's `make install-dev` downgraded the governor, and which builds are outside the cap. Run it before concluding your own build is slow or hung — a build whose rustc sits at `Compiling` for minutes with no CPU is queued, not broken. A foreground Bash call whose cargo goes silent at `Compiling` is the same thing: relaunching it puts the new cargo at the BACK of the queue.
 
 ## Performance — reuse the cache, clamp the QoS, cap the jobs
 

.claude/skills/rust-build/SKILL.md

@@ -35,7 +35,7 @@ It prints which target dir it chose and why, then execs `cargo` with `CARGO_TARG
 - **QoS clamp (darwin only):** cargo runs under `taskpolicy -c utility`, so interactive work preempts fleet builds; an uncontended build still gets all cores. `NUB_BUILD_FG=1` opts out.
 - **Job cap on big hosts (>8 cores):** `CARGO_BUILD_JOBS = ncpu-4` unless the caller already chose (pre-set `CARGO_BUILD_JOBS`, `NUB_BUILD_JOBS`, or an explicit `-j`/`--jobs` — cargo's flag outranks the env var).
 
-These cover builds going THROUGH the wrapper (or make). Direct `cargo` invocations are clamped by a machine-global control: `make qos-global` installs `scripts/rustc-qos.sh` as the cargo `rustc-wrapper` in `~/.cargo/config.toml`, so every rustc on the host compiles at utility QoS (`install-dev` re-runs it, so it self-heals). Same `NUB_BUILD_FG=1` opt-out; toggling it does not invalidate fingerprints.
+These cover builds going THROUGH the wrapper (or make). Direct `cargo` invocations are clamped by a machine-global control: `make qos-global` installs `scripts/rustc-qos.sh` as the cargo `rustc-wrapper` in `~/.cargo/config.toml`, so every rustc on the host compiles at utility QoS, at most TWO builds compile at a time (the rest queue first-come-first-served; `make build-status` shows the holders and the queue), and the compiling builds share `NUB_RUSTC_LIMIT` (default 6) rustc tokens (`install-dev` re-runs it, so it self-heals). `NUB_BUILD_FG=1` opts a HUMAN's foreground build out of the clamp and the queue (and, through this wrapper, out of the tokens too) — never set it from an agent; `NUB_BUILD_SLOTS=0` in a build's environment disables only the slot layer for that build. Toggling any of these does not invalidate fingerprints. Details: the `rust-build-hygiene` skill.
 
 ## Why one shared target dir
 

.githooks/pre-push

@@ -213,7 +213,7 @@ fi
 # and runs only when the pushed range touches tests/cross-runtime/**. Like the
 # site gate, an undecidable range runs the check rather than skipping it.
 #
-# The push refs on stdin are read ONCE here and replayed to the site gate below.
+# The push refs on stdin are read ONCE here and replayed to every gate below.
 # Bypass: NUB_SKIP_README_TABLE_CHECK=1 git push   ·   git push --no-verify
 PUSH_REFS=$(cat)
 if [ -z "${NUB_SKIP_README_TABLE_CHECK:-}" ] && [ -n "$SKILLS_ROOT" ] && [ -f "$SKILLS_ROOT/tests/cross-runtime/readme-table.mjs" ]; then
@@ -250,6 +250,25 @@ PUSHREFS
   fi
 fi
 
+# A change to the machine-global rustc governor must bump its version stamp, or
+# build-status can no longer tell a stale installed copy from a current one.
+# The base is the MERGE-BASE with trunk, not the branch's previous push — the
+# stamp names a version of the protocol, so CI's build-slots job wants one bump
+# per pull request and this gate must want the same, not one per push.
+# Bypass: NUB_SKIP_QOS_VERSION_CHECK=1 git push
+#
+if [ -z "${NUB_SKIP_QOS_VERSION_CHECK:-}" ] && [ -n "$SKILLS_ROOT" ] \
+  && [ -x "$SKILLS_ROOT/scripts/rustc-qos-version-check.sh" ]; then
+  while read -r _local_ref local_sha _remote_ref remote_sha; do
+    case "$local_sha" in *[!0]*) ;; *) continue ;; esac
+    base=$(git merge-base "$local_sha" origin/main 2>/dev/null) || base=""
+    [ -n "$base" ] || continue
+    (cd "$SKILLS_ROOT" && scripts/rustc-qos-version-check.sh "$base" "$local_sha") || exit 1
+  done <<PUSHREFS
+$PUSH_REFS
+PUSHREFS
+fi
+
 [ "${NUB_SKIP_SITE_CHECK:-}" = "1" ] && exit 0
 
 ROOT="$SKILLS_ROOT"

.github/workflows/ci.yml

@@ -1378,6 +1378,42 @@ jobs:
           persist-credentials: false
       - run: make version-check
 
+  # The build-slot layer of the machine-global rustc governor (scripts/rustc-qos.sh)
+  # is a lock-free protocol over a directory tree whose failure mode is a host-wide
+  # stall, and nothing else in CI runs scripts/*.sh. The harness drives the real
+  # script with a fake cargo/rustc pair — POSIX sh plus a C compiler, ~3 min — and
+  # the version check refuses a governor change that forgot to bump the stamp
+  # build-status relies on to spot a stale installed copy.
+  build-slots:
+    name: Build-slot governor (${{ matrix.os }})
+    needs: matrix-plan
+    # Path-gated on BOTH groups: the governor itself is scripts/**, which lives
+    # in the `scripts` filter, while the harness is tests/**, which lives in
+    # `rust` — neither group alone covers this job. Same shape as docs-links.
+    if: needs.matrix-plan.outputs.run_rust == 'true' || needs.matrix-plan.outputs.run_scripts == 'true'
+    # Both platforms: the governor is only ever INSTALLED on the maintainer's
+    # darwin host (bash-3.2 /bin/sh, BSD ps/awk), while ubuntu's dash + procps
+    # is the other dialect the same POSIX script must keep speaking.
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-14]
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd  # v5.0.1
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+      - name: Version stamp bumped with the governor
+        if: github.event_name == 'pull_request'
+        env:
+          BASE: ${{ github.event.pull_request.base.sha }}
+        run: scripts/rustc-qos-version-check.sh "$BASE" HEAD
+      - run: tests/build-slots/run.sh
+
   # Make THIS job (`CI gate`) the sole required status check in branch protection.
   # A docs-only PR → Rust matrix skipped → this gate green → mergeable. A Rust PR
   # → matrix runs → this gate reflects it. The merge gate is never weakened: a
@@ -1406,7 +1442,7 @@ jobs:
   ci-gate:
     name: CI gate
     if: always()
-    needs: [matrix-plan, check, clippy, embed-runtime, windows-embed-roundtrip, test, pnp, fmt, types-fixtures, docs-links, site-build, oxc-lockstep, version-consistency, target-gc]
+    needs: [matrix-plan, check, clippy, embed-runtime, windows-embed-roundtrip, test, pnp, fmt, types-fixtures, docs-links, site-build, oxc-lockstep, version-consistency, target-gc, build-slots]
     runs-on: ubuntu-latest
     steps:
       - name: Verify no required job failed

Makefile

@@ -25,7 +25,7 @@ else
   CARGO_FLAGS = --profile $(PROFILE)
 endif
 
-.PHONY: build addon addon-fast install-dev uninstall-dev qos-global build-status target-gc test-target-gc test verify test-node-matrix bench clean npm-build npm-publish npm-publish-dry
+.PHONY: build addon addon-fast install-dev uninstall-dev qos-global build-status build-slots-off build-slots-on target-gc test-target-gc test test-build-slots verify test-node-matrix bench clean npm-build npm-publish npm-publish-dry
 
 build: addon
 	$(CARGO) build $(CARGO_FLAGS)
@@ -87,16 +87,19 @@ addon-fast:
 	@echo "Built: runtime/addons/nub-native.node (fast profile)"
 
 # Machine-global rustc governor: every cargo invocation on this host — any
-# worktree, clone, or direct `cargo` call — compiles at utility QoS AND takes a
-# token from a global concurrency semaphore, closing the gap the entry-point
-# clamps above leave open. The QoS half protects interactive work; the semaphore
-# half is what stops N concurrent agent builds from multiplying their individually
-# polite job caps into an N-fold oversubscription. See scripts/rustc-qos.sh.
+# worktree, clone, or direct `cargo` call — compiles at utility QoS, waits its
+# turn for a build slot (at most two builds compile at once; the rest queue),
+# AND takes a token from a global rustc semaphore, closing the gap the
+# entry-point clamps above leave open. The QoS half protects interactive work;
+# the slot is what stops N concurrent agent builds from multiplying their
+# individually polite job caps into an N-fold oversubscription and a swap storm.
+# See scripts/rustc-qos.sh.
 qos-global:
 	@scripts/qos-global.sh
 
-# Why is this machine saturated? Prints the SUM no single session can see: load,
-# live builds, semaphore occupancy, and which builds are outside the cap.
+# Why is this machine saturated, and why is my build not starting? Prints the
+# SUM no single session can see: load, which builds hold the compile slots and
+# who is queued behind them, token occupancy, and which builds are outside the cap.
 build-status:
 	@scripts/build-status.sh
 
@@ -108,13 +111,28 @@ target-gc:
 test-target-gc:
 	@tests/target-gc/run.sh
 
+# Host-wide emergency switch for the build-slot layer: every wrapper checks the
+# file on entry and on each wait iteration, so `off` releases builds already
+# queued. Leaves the QoS clamp and the rustc tokens in place.
+build-slots-off:
+	@mkdir -p $${NUB_BUILD_SEM_DIR:-$$HOME/.cache/nub/build-sem} && touch $${NUB_BUILD_SEM_DIR:-$$HOME/.cache/nub/build-sem}/off && echo "build slots OFF (make build-slots-on to restore)"
+build-slots-on:
+	@rm -f $${NUB_BUILD_SEM_DIR:-$$HOME/.cache/nub/build-sem}/off && echo "build slots ON"
+
 uninstall-dev:
 	rm -f $(BIN_DIR)/nub-dev $(BIN_DIR)/nubx-dev
 	@echo "Removed nub-dev and nubx-dev from $(BIN_DIR)"
 
 test:
 	$(CARGO) test
 
+# The build-slot layer of the machine-global rustc governor, driven by a fake
+# cargo/rustc pair in a private state dir — ~3 min, no real build, never touches
+# the installed governor. CI runs the same script (the `build-slots` job, on
+# ubuntu and macOS), and `verify` includes it.
+test-build-slots:
+	@tests/build-slots/run.sh
+
 # Bounded host-local gate. Platform matrices, Docker jobs, and change-specific
 # end-to-end tests remain separate parts of the pre-push verification loop.
 verify:
@@ -138,6 +156,7 @@ verify:
 	@# above. A lint ships nothing, so grant it here exactly as ci.yml's clippy job does
 	@# rather than vendoring npm packages into runtime/ on every developer's tree.
 	NUB_ALLOW_INCOMPLETE_RUNTIME=1 NUB_SHARED_TARGET="$(CURDIR)/target" "$(RUST_BUILD)" clippy --all-targets --all-features --profile fast -- -D warnings
+	@tests/build-slots/run.sh
 	(cd crates/nub-native && NUB_SHARED_TARGET="$(CURDIR)/target" "$(RUST_BUILD)" clippy --all-features --profile fast -- -D warnings)
 	tests/brand-lint/check-env-reads.sh
 	tests/brand-lint/check-path-literals.sh