build: cap the host at two compiling builds over six rustc tokens

[colinhacks]

The rustc-count cap still let 10 rustc run 7 LLVM threads each at 1-3 GiB apiece. This adds a build-level layer: at most NUB_BUILD_SLOTS (default 2) cargo invocations compile at once machine-wide, FIFO, sharing NUB_RUSTC_LIMIT (default 6) rustc tokens. Slot = outermost cargo ancestor; reclaimed when the holder dies or idles NUB_BUILD_IDLE (120s), place kept on re-queue; rust-analyzer exempt; fail-open in place (wiped state dir, dead cargo, make build-slots-off); one compound command, so an older installer's copy cannot truncate a running wrapper; a build queued 20s says so once.

qos-global installs by rename; build-status shows holders, queue, stale installs.

Fake-cargo harness (15 scenarios, mutation-checked, ubuntu + macOS CI) and live on the host.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nub	Ready	Preview	Aug 31, 2026 10:43pm

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[pullfrog]

Important

NUB_BUILD_FG=1 no longer opts a build out of the global governor, but this PR's doc edit says it does. A latency-sensitive foreground build's workspace crates now queue behind the agent fleet for up to NUB_BUILD_WAIT (3600s).

Reviewed changes — a new machine-wide build-slot layer in the global cargo rustc-wrapper, plus the status/installer/doc changes around it.

• Build slots in rustc-qos.sh — at most NUB_BUILD_SLOTS (default 1) cargo invocations may be compiling at once host-wide, FIFO. State is a directory tree under ~/.cache/nub/build-sem (slot/<n>/{pid,stamp,active/<pid>}, queue/<cargo-pid>), with mkdir as the atomic claim and a rename-then-rm -rf retire so two racing waiters cannot double-free.
• Build identity via process-ancestry walk — one ps snapshot plus an awk walk picks the outermost cargo ancestor, so nested cargo inherits rather than deadlocks; a rust-analyzer ancestor exempts the build. Cached per parent pid, keyed on the parent's start time to defeat pid recycling.
• Idle reclaim — a slot is reclaimable once no compile is live and none has started for NUB_BUILD_IDLE (90s), so cargo test running tests or a cargo blocked on a target-dir lock does not hold the machine.
• qos-global.sh write-then-rename — replaces cp over the live wrapper, which would have corrupted every wrapper mid-read machine-wide.
• build-status.sh slots section — shows the holder, per-slot live compile count, and the queue.
• Skill docs + Makefile comment rewritten to describe three jobs instead of two.

I verified the ancestry walk myself against known-answer cases (no-cargo → fail-open, cargo → its pid, rust-analyzer→cargo → exempt, nested → outermost) and it is correct on both ps -o comm= shapes. I also confirmed empirically on sh, bash, dash and bash --posix that the new trailing [ -n "$_bslot" ] && rm -f … in the EXIT trap does not clobber rustc's exit status through exit $? — that one is clean.

⚠️ A concurrency state machine with no committed test

The PR body cites "a fake-cargo harness (8 scenarios)" that verified retire/reap/queue behavior, but the harness is not in the diff. This is precisely the code where a throwaway probe is worth the least: it is a lock-free protocol over a directory tree, its failure mode is a host-wide stall rather than a wrong answer, and it is exec'd hundreds of times per build in front of every compile on the machine. tests/ already holds ~35 per-feature harnesses and AGENTS.md asks for exactly this promotion; nothing in CI or .githooks/ currently lints or exercises scripts/*.sh at all.

Technical details

# Promote the fake-cargo harness into the tree

## Affected sites
- PR description — describes 8 verified scenarios; none are in the diff.
- `scripts/rustc-qos.sh:105-277` — new lock-free protocol, no test coverage.
- `tests/` — ~35 sibling harnesses (`tests/pnp/`, `tests/launcher/`, …) establish the convention.

## Required outcome
- The 8 scenarios run reproducibly from the repo, without a 10-core mac or a real cargo.
- The harness proves it can FAIL: at minimum a positive control showing a slot is
  genuinely withheld, so a future refactor that silently disables the layer (e.g. the
  ancestry walk returning empty and failing open) goes red instead of green.

## Suggested approach (optional)
- `tests/build-slots/` with a `run.sh`: a fake `cargo` shell script on `PATH` that does
  NOT `exec` (an `exec`ing parent is not an ancestor and the walk correctly finds
  nothing — this silently turns the whole layer off), a dummy `rustc`, and
  `NUB_BUILD_SEM_DIR` / `NUB_BUILD_IDLE` / `NUB_BUILD_WAIT` set small so the matrix runs
  in seconds. It is POSIX sh, so it runs on the Linux CI legs, not just darwin.
- Worth covering explicitly: FIFO order across three builds; idle reclaim then re-queue;
  holder SIGKILLed mid-compile; a queued wrapper killed while its cargo survives.

## Open questions for the human
- Is a `shellcheck` gate over `scripts/*.sh` worth adding at the same time? There is none
  today, though several scripts already carry `# shellcheck disable` pragmas.

ℹ️ The cap does not hold until every host re-runs qos-global, and build-status reports those builds as governed

A build running through a pre-PR ~/.cargo/rustc-qos.sh creates no slot dir and takes no ticket, so it compiles freely alongside the slot holder — the serialization silently does not hold. That is expected during rollout, but build-status.sh's per-cargo verdict only distinguishes a blanked RUSTC_WRAPPER (partial (workspace)) from everything else (governed), so a stale-wrapper build is labelled governed while being outside the new layer entirely. Given qos-global.sh:23-32 records that 58 of 61 live worktrees carry a stale installer that cps the old wrapper back, this state is likely rather than hypothetical.

Technical details

# Make a stale wrapper visible

## Affected sites
- `scripts/build-status.sh:78-81` — `case "$w"` marks anything that is not an explicitly
  empty `RUSTC_WRAPPER` as `governed`, with no notion of wrapper vintage.
- `scripts/qos-global.sh:23-32` — documents that stale worktrees re-install the old wrapper.

## Required outcome
- `make build-status` distinguishes "governed by a wrapper that implements build slots"
  from "governed by an older wrapper that does not", so the first question after an
  unexplained load spike has an answer.

## Suggested approach (optional)
- Stamp a version into the wrapper (a `# rustc-qos-version: N` line) and have
  `build-status.sh` grep the installed copy, reporting `stale wrapper — run make
  qos-global` when it is missing or older.
- A cheaper proxy: a cargo that has been alive for longer than a few seconds while
  holding no slot and no queue ticket is outside the layer.

ℹ️ Nitpicks

• The throughput justification is repeated in three files as an absolute — "Serialising builds costs no throughput (N builds sharing 10 cores take at least as long as N builds in series)". That holds only while each build saturates every core; real cargo builds have serialization points (dependency-graph bottlenecks, final link, single-crate codegen) where a sibling build would have used the idle cores. The memory argument for serializing is strong on its own, so consider softening to something like "costs little throughput" rather than stating a strict inequality that does not hold in general. Sites: scripts/rustc-qos.sh:21-23, .claude/skills/rust-build-hygiene/SKILL.md:51.
• NUB_BUILD_SLOTS=0 is the only genuine way to bypass the new layer, and it is documented only in the rustc-qos.sh header — neither skill mentions it, so an agent or maintainer reading the skills has no escape hatch to reach for.

---

Two specialist investigations I dispatched — on whether an orphan queue/<cargo-pid> ticket can wedge the head of the queue until the 3600s ceiling, and on whether a recycled pid behind a stale active/<pid> marker can pin a slot — did not report back before this review was submitted. Neither was proven reachable, and neither is reflected in the findings above; both are livelock-shaped and worth settling before this is treated as load-bearing on the dev host.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Important

Two issues worth fixing before this ships. first/<cargo-pid> is never cleaned up on the happy path, so it grows without bound and a recycled pid inherits an ancient first-queued time and sits permanently at the head of the queue — reproduced. And three of the four numeric tunables are unvalidated, so a NUB_BUILD_WAIT=1h silently disables the fail-open ceiling this script's header promises — also reproduced, A/B.

All three threads from the previous review are addressed and resolved: the NUB_BUILD_FG slot exemption (rustc-qos.sh:177 + rust-build.sh:272), the build-status.sh ticket-read race, and the doc scoping in rust-build/SKILL.md. The body-level asks landed too — the harness is committed, and # rustc-qos-version: 2 plus the STALE WRAPPER check finally makes a stale installed copy visible. The two hardening fixes in this commit both hold up under test: the heartbeat prune does clear an orphaned ticket, and NUB_BUILD_MAXCOMPILE does stop a recycled pid from pinning a slot forever.

ℹ️ Informational

The harness has no runner. tests/build-slots/run.sh is referenced from no Makefile target, no CI workflow, and no README — 26 of 40 dirs under tests/ carry one; this one does not. Nothing in the repo lints or exercises scripts/*.sh at all, so the positive control that was asked for only runs when someone remembers to type the path. Relatedly, # rustc-qos-version: 2 is hand-maintained with nothing enforcing a bump, so the first semantics change that forgets it leaves every stale host reported as current — which is the exact failure the stamp was added to prevent.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Important

One regression to fix before this ships. The rewritten first/ read at rustc-qos.sh:280 moved its 2>/dev/null behind the input redirect, so every build now prints cannot open …/sem/first/<pid>: No such file to the wrapper's real stderr — 21 of them across 18 builds in one harness run, reproduced. Same class as the Illegal number: 5s spew this commit fixed.

Reviewed changes — 9b14b49 against the previously reviewed 4b9881b, 11 files, 3 commits.

All five prior threads are genuinely addressed, and I re-derived each rather than taking the diff's word for it: the three -ge 0 tunable fallbacks (rustc-qos.sh:132-134), the *.* mid-rename guard on _reap's queue loop, dead-pid GC for first/ plus an lstart line so a recycled pid cannot inherit a place, the NO SLOT (fg/stale?) heuristic removed, and the harness assertion flipped to -le 4 with NUB_BUILD_MAXCOMPILE unset alongside the others. Threads replied to and resolved.

I ran tests/build-slots/run.sh on a 4-core ubuntu runner: all 8 scenarios pass, 95s wall, rc=0. The new required gate is green and its timing margins hold here.

⚠️ The harness went green while the script errored on every single build

That same 95s run emitted 21 cannot open diagnostics and still printed build-slots: all scenarios passed. The scenarios assert only on the timeline they parse from the fake rustc; nothing inspects what the wrapper writes to stderr. Now that build-slots is in ci-gate's needs, that blind spot is the last thing between this defect class and main — and it is not hypothetical, since both stderr-noise bugs found on this PR (Illegal number: 5s last round, cannot open this round) were invisible to it.

Capturing the wrapper's stderr across the whole run and failing on anything the scenarios did not deliberately provoke would close it in one assertion, independent of the eight scenarios.

Two smaller wiring notes on the same job: it joins ci-gate with no path filter, so a docs-only PR pays the ~95s, and with no timeout-minutes, so a wedged scenario burns the job default instead of failing fast.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Important

The new path gate on the build-slots job skips it for the exact change class it guards. scripts/** is not in the rust paths-filter group, so a PR that touches only scripts/rustc-qos.sh runs neither the harness nor the version-stamp check, and ci-gate still goes green.

Reviewed changes — f946b93 against the previously reviewed 9b14b49.

• The first/ read no longer leaks to stderr — 2>/dev/null now precedes the input redirect at rustc-qos.sh:283, and the same reordering was applied to the two sibling reads (_pcache at :154, the ticket heartbeat at :237). No mis-ordered } < shape remains anywhere in scripts/*.sh.
• Added a whole-run stderr assertion to the harness — every wrapper invocation appends its stderr to $T/log/wrapper.err, and a scenario-independent check fails the run if that file is non-empty.
• Rewired the FG tell in build-status.sh — :102-106 now treats a blank RUSTC_WORKSPACE_WRAPPER= paired with a blank RUSTC_WRAPPER= as the rust-build.sh foreground signature, keeping NUB_BUILD_FG=1 for the make build and bare-cargo paths.
• Aligned the pre-push gate with CI — .githooks/pre-push:217 now always uses git merge-base "$local_sha" origin/main, and NUB_SKIP_QOS_VERSION_CHECK=1 is an escape hatch alongside the lat and site gates.
• Bounded and path-gated the CI job — timeout-minutes: 10, plus needs: matrix-plan and an if: on run_rust.
• Widened serialize's pair-check bound from -le 4 to -le 5.

I ran tests/build-slots/run.sh end to end on this branch: all 8 scenarios plus the new stderr assertion pass, rc=0, and wrapper.err is empty — the cannot open regression is genuinely gone rather than relocated. That run also measured last A start at exactly 4 in serialize, so the previous bound had zero margin; the widening is a justified tolerance change, not a masked failure. I separately confirmed the new capture has no hole: all six $W call sites in the harness carry the redirect, reset() never truncates wrapper.err, and nothing is still writing when the assertion reads it.

ℹ️ The now-required gate rests on absolute wall-clock bounds

build-slots sits in ci-gate's needs, so a timing miss on a contended runner paints the sole branch-protection check red. Most assertions are relations between logged events (at B start >= at A cargo-end) and are robust to a slow runner, but serialize's pair check and idle's gap check are absolute-second bounds — and serialize's measured exactly at its old limit this run, which is what forced the widening in this commit. Nothing is broken today; the exposure is that the next slow runner tightens the same screw again.

Technical details

# Absolute-second assertions on a required CI gate

## Affected sites
- `tests/build-slots/run.sh:87` — `[ "$(last A start)" -le 5 ]`. Measured 4 on a 4-core
  ubuntu runner at `f946b93`; correct behavior is 3, over-throttled behavior is ~9.
- `tests/build-slots/run.sh:124` — `[ "$(last C start)" -le $(( $(at C start) + 7 )) ]`,
  three 3s compiles under `NUB_BUILD_IDLE=2`.

## Required outcome
- The gate stays green on a runner slower than the one these bounds were tuned on,
  without a future contributor widening the constant again each time.

## Suggested approach (optional)
- Where a relation expresses the same claim, prefer it: `serialize`'s "ran in pairs" is
  really "the last start is closer to the first start than a serial run would allow",
  which can be stated against `at A cargo-end` minus one compile duration rather than
  against a literal.
- Where a literal is unavoidable, widen it toward the broken value's midpoint rather
  than by one second, and put the correct/broken pair in a comment so the next person
  can see how much slack is left.

## Open questions for the human
- Is `build-slots` intended to stay in `ci-gate`'s `needs` long-term, or was that a
  rollout convenience? A timing-shaped harness is a different risk profile as a required
  check than as a `make` target.

ℹ️ Nitpicks

• The PR description says the slot is reclaimed after NUB_BUILD_IDLE (90s); the default is 120s in rustc-qos.sh:126 and in both skill docs.
• build-status.sh:88-91 still justifies the partial (workspace) label with "qos-global also registers the wrapper as rustc-workspace-wrapper, which those callers do not blank". rust-build.sh:272 now blanks both keys, so that sentence no longer describes any in-tree caller — the label is reachable only from a stale checkout or a hand-rolled config.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 12:40am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 12:40am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 12:40am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 12:40am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

ℹ️ The torn-read defect this commit exists to fix is not covered by the harness. I reverted _stamp/_mark to the pre-fix in-place > writes and every slot-contention scenario stayed green. Nothing here is broken; the gap is that a future regression would land green through a required check.

Reviewed changes — 4c1b5ab against the previously reviewed dad1ec4.

• Write-then-rename for slot state — _stamp/_mark (rustc-qos.sh:234-235) build a temp and mv it into place, so a reader can no longer land between a >'s truncate and its write. queue/ and first/ got the same treatment at :372-376. The dot-prefixed and .$$-suffixed temps are invisible to the active/* glob and to the *.* case guards in _sweep, _head_of_queue, _say_queued and build-status.sh:74 — checked each. The three remaining plain > writes ($_mx/pid, slot/<n>/pid, the token pid) are all into a file that has just been created by a winning mkdir, so there is no truncate to tear.
• Reader-side reordering — _reap scans active/* first and reads stamp only when it finds none live; _release writes the stamp first and drops its marker second. On a local filesystem one writer's rename()-then-unlink() is observed in that order, so "marker gone implies fresh stamp already there" holds — this closes the window rather than narrowing it.
• Fallback directions — an unreadable marker age with a live pid counts as fresh (:265), and _reap skips the whole poll when date could not fork (:251), instead of judging every age against an empty _tnow.
• _hold splits its failures — 1 (lost the slot, keep waiting) vs 2 (table unwritable, fail open), and _release now stamps only a slot whose pid still names our own cargo.
• NUB_RUSTC_SEM_HELD=1 is exported unconditionally — I checked whether this can now let an inner workspace-wrapper hop skip a protocol it should have run. It cannot: the token block at :481-514 runs regardless of slot exemption, so the outer shell always makes the definitive token decision before the export. The one path that skips everything (the -vV/--print probe exec at :119) also skips the export, and the inner re-runs that same check independently. This is more correct than the old conditional, not less.
• Harness — fifo's B now compiles 2s so arrival order alone cannot satisfy the check; new deadcargo scenario; three new idle assertions; kill 10→12, serialize 5→6, idle +7→+9 and +14→+15.

I ran the harness on a 4-core ubuntu runner and mutation-checked every assertion added or changed in this commit:

assertion	measured	bound	mutation	result
deadcargo — Q ran before A ended	Q t+3, A end t+8	relation	drop the kill -0 "$_cargo" guard at :422	red — Q never starts
idle — A's re-queue took the slot promptly	last A start 11	14	make _ticket forget the persisted first/ time	red — 43
idle — D never ran before C's cargo exited	34 vs 34	relation	same	red
idle — A's second compile ran before D	11 < 34	relation	same	green (weaker)

fifo, kill, serialize, twoslots, orphan and nested also pass, and wrapper.err is empty across every run. The version stamp is at 6 and consistent between rustc-qos.sh:2, build-status.sh:90 and the check script; defaults are NUB_BUILD_SLOTS=2 / NUB_RUSTC_LIMIT=6 / NUB_BUILD_IDLE=120, matching the title and body. The prior round's build-status.sh wording nitpick is gone.

ℹ️ The harness cannot fail on the bug this commit fixes

The headline change is the torn-read fix — an empty stamp read as "stampless" retired a live slot, reproduced at 30% per the commit. I reverted _stamp and _mark to their pre-fix in-place > form and ran the six scenarios that actually contend for a slot:

idle       ok (x2)    serialize  ok    orphan  ok
deadcargo  ok         twoslots   ok    nested  ok
build-slots: all scenarios passed

The reader-side defenses are unexercised for the same reason: no scenario ever produces an empty stamp or an empty active/<pid>, so flipping the :265 fallback direction or undoing _reap's marker-first ordering would also land green. build-slots is in ci-gate's needs, so this is the class of regression that reaches main unannounced.

A race is hard to schedule, but the state it produces is trivial to plant, and NUB_BUILD_SEM_DIR already points at the harness's own tree (run.sh:69). A scenario that lets A take slot/1, then does : > $T/sem/slot/1/stamp and : > $T/sem/slot/1/active/<pid> while A is mid-compile, and asserts B does not steal the slot, is deterministic and needs no timing at all. It would go red under either the pre-fix reader ordering or a flipped fallback.

ℹ️ Nitpicks

• This commit widens four absolute-second bounds (run.sh:96 5→6, :117 10→12, :143 +7→+9, :146 +14→+15) and adds a fifth at :149. I measured real slack in all of them on this runner (3–8s) and mutation-confirmed they still go red, so nothing is masked — but this is the second consecutive commit turning the same screw on a required check, which is the open question from the last round rather than a new finding.
• The PR body says "12 scenarios"; there are 13.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

ℹ️ No critical issues — minor suggestions inline.

Incremental review of the three commits since the last Pullfrog review at 4c1b5ab.

Reviewed changes

• One validated clock reading per poll (rustc-qos.sh:413-429). _tnow is read and range-checked once at the top of the loop, then reused by _reap, _ticket, _say_queued, the claim path and the wait ceiling. I traced every read and write of _tnow: only two writes exist (:423 and _release's own at :565), both validated before use, and every helper that consumes it is reachable only from the validated loop or recomputes its own. The redundant internal guards in _reap are correctly gone.
• _hold marker leak fixed (:342-357) — retires the open thread from the last round. The marker is dropped on both failure branches, and fail-open is gated on the slot's pid still naming this build.
• Broken pipe suppression on _now() and the harness's fake rustc, with build.sh's own date deliberately left loud since its stderr does not feed wrapper.err. Consistent with the whole-run stderr assertion.
• New torn scenario (run.sh:199-211) — the first coverage for the reader-side torn-read defenses. Mutation-verified live on this run: blanking the marker's write-time fallback at rustc-qos.sh:265 (_w=$_tnow → _w=0) turns it red (B starts at t+2 instead of t+8). Clean run passes with wrapper.err empty.
• idle restructured around the new build pre-compile-delay argument, trading two absolute-second bounds for one relational place-keeping check (run.sh:157). Mutation-verified: making _ticket forget the persisted first/ time fails three of its assertions. Measured slack on a 4-core ubuntu runner is 3–5s on each remaining bound. This is the first commit in three that reduces the count of absolute-second assertions on the required gate — the right direction.
• Version stamp 6 → 8; PR body's scenario count now matches the harness's 14.

Note, not a finding

torn plants the post-torn state rather than producing it, so the write-side _stamp/_mark write-then-rename fix is still uncovered by construction. That is inherent to the remedy suggested last round, not a gap introduced here — the reader-side defenses are what became testable, and they now are.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Incremental review of the two commits since the last Pullfrog review at 2e621e8. Both of that review's findings are fixed, and the delta introduces nothing new.

Reviewed changes

• _noclock is now consecutive (rustc-qos.sh:430). _noclock=0 after a validated reading, exactly as suggested. The counter now means what its comment says — a minute of consecutive clock failures, not a lifetime tally that intermittent date misses could walk to 60 over a long queue wait and silently drop the slot cap.
• torn has a positive control (run.sh:206,211). The plant counts the markers it truncates and asserts n > 0. Mutation-verified live on this run: pointing the glob at slot/9 makes it a no-op and the new assertion goes red (FAIL A held slot 1 with a live marker to truncate, rc=1). It is a real gate, not a green-either-way line. check() evals in the current shell so n is in scope, and set -u makes an unset one fail rather than pass — the failure direction is safe. Clean run passes with wrapper.err empty.
• 2>/dev/null on uname and the ancestor-walk awk (:124, :211), completing the invariant the rewritten comment claims. I checked the claim rather than taking it: every $( in the file is either arithmetic $(( … )) or one of uname :124, ps -o lstart= :195/:367, the ps | awk pipeline :201-211, find :283, and $(_now) :424/:567 — the last silenced inside _now itself. All six silence their stderr, so "EVERY command substitution" holds as written. This is what keeps the harness's whole-run stderr assertion meaningful when a wrapper is SIGKILLed mid-substitution.
• Version stamp 8 → 10, one bump per commit. Both consumers sed-extract it (build-status.sh:90, rustc-qos-version-check.sh:19-29), so nothing hardcodes the number and no companion edit is owed.

Still uncovered, not a blocker: the _noclock fail-open branch itself has no scenario — it is now the one path in the wait loop with no test, and covering it would need clock-failure injection into the harness's fake date. Worth a follow-up rather than this PR.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

ℹ️ No critical issues — minor suggestions inline.

Reviewed changes

One commit since the last review (a5e5dda): e56d815, touching tests/build-slots/run.sh only (+16 lines). It adds a noclock scenario that starves the wrapper of a clock by putting a failing date first on PATH, and switches the fake cargo/rustc to /bin/date so the event log the assertions read stays populated. That /bin/date split is the right call and worth the comment it carries.

I ran the harness rather than reasoning about it. noclock passes in 61s (t+0s N cargo-start; t+59s N start; t+60s N end), stderr clean. Mutating the threshold [ "$_noclock" -ge 60 ] → -ge 6 at scripts/rustc-qos.sh:431 turns it red (t+5s N start, rc=1), so the scenario is a genuine gate and not theatre.

I also timed the full suite, since run.sh:12 now claims "~5 min" against a timeout-minutes: 10 required check. Measured 3m40s on a 4-core ubuntu runner across four batches (34.5s + 76.4s + 46.8s + 61.3s), all 15 scenarios green. ~2.7x margin, so the timeout is fine — but see the nitpick about where that estimate is recorded.

One coverage gap noted inline, non-blocking.

ℹ️ Nitpicks

• run.sh:221 and :223 say "measured ~70s" and "(~60-70s)", but 70s isn't reachable: the 60th poll breaks before sleeping, so there are exactly 59 sleeps by construction and the run measures 59s. The -ge 55 lower bound therefore has 4s of slack against a hard floor rather than the ~15s the comment implies. Worth restating as "59 sleeps, measured 59s" so a future reader doesn't loosen the bound on the strength of a wrong number.
• The harness runtime is stated in three places and only one moved. run.sh:12 now says "~5 min"; Makefile:122-124 still says "~3 min" and .github/workflows/ci.yml:1384-1385 still says "~2 min". Measured 3m40s.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Incremental review of the one commit since the prior Pullfrog review at e56d815: 79b9eb1, touching .github/workflows/ci.yml, scripts/rustc-qos.sh and tests/build-slots/run.sh. It closes last round's coverage finding exactly as suggested, and both nitpicks with it.

Reviewed changes

• Made the clock-miss threshold a tunable. NUB_BUILD_NOCLOCK_MAX (default 60) at rustc-qos.sh:156, consumed at :431, listed in the header's tunable block, and unset alongside its siblings at run.sh:78 so a host value cannot leak into the harness. Its validation is -ge 1 rather than the siblings' -ge 0, which is the right call — a 0 threshold would fail open on the very first miss and silently disable the layer. Version stamp 10 → 11.
• noclock now pins the CONSECUTIVE semantics. The scenario splits in two: a permanently-failing date with NUB_BUILD_NOCLOCK_MAX=5 for the threshold, plus a new flipclock date that fails every other call while A holds the slot for 12s — so cumulative counting would trip the valve around poll 5 while consecutive counting never gets past one.
• Reconciled the runtime estimates. run.sh:12 and ci.yml:1384 now both say "~3 min", matching Makefile:122-124.

I ran the scenario rather than reasoning about it. Clean: t+4s N start for the threshold case, and t+13s B start after A's t+13s end for the consecutive case — rc=0, wrapper.err empty, 20.3s where the old scenario took 61.3s. That puts the suite near 3 min, so the three runtime comments are now accurate rather than merely consistent.

Mutation-verified: reverting _noclock=0 at rustc-qos.sh:434 to a no-op — the exact revert that stayed green last round — now turns the second case red (t+5s B start, B failing open onto A's live compile; 2 assertions red, rc=2). Restored and confirmed with diff -q. The gap is genuinely closed, not reworded.

Two things I checked rather than assumed. The -ge 4 lower bound has no nominal slack but cannot flake low: four sleep 1s make elapsed ≥ 4.0s, so the integer-second difference is ≥ 4 by construction — unlike the old -ge 55 sitting one nudge under a 59s floor. And [ "$(cat "$T/flipclock/n")" -ge 6 ] is a genuine positive control, not decoration: without it, a PATH prefix that failed to reach the wrapper would make "B waited for A" trivially green.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes — every commit SHA moved, but the delta since the last review (79b9eb1) is a pure rebase onto main's a247a22 (self-pruning target-dir GC). Net PR content change is zero: the tree diff 79b9eb1..d0c97cf touches only 4 of the PR's 13 files, and a path-scoped diff shows those 4 changed only by main's content.

Byte-identical to the reviewed revision: scripts/rustc-qos.sh (still # rustc-qos-version: 11), tests/build-slots/run.sh, scripts/build-status.sh, scripts/qos-global.sh, scripts/rustc-qos-version-check.sh, and all four SKILL.md files.

What I checked on the rebased tree, since a rebase is where hunks get silently dropped and where two new subsystems can collide:

Check	Result
PR hunks survived	wrapper_off in rust-build.sh, the pre-push qos-version gate, build-slots-off/-on + test-build-slots + verify in the Makefile — all present
main's hunks survived	.PHONY keeps target-gc test-target-gc; main's Makefile targets and @tests/target-gc/run.sh in verify appear as unchanged context
ci-gate needs	carries both target-gc and build-slots; ci.yml parses, 17 jobs, identical run_rust || run_scripts gate on each
GC vs. slot state	target-gc.sh candidates are strictly shared-target / shared-target-* plus worktree target/ dirs — ~/.cache/nub/build-sem is never collected
GC probes vs. the wrapper	rust-build.sh --print-target exits at :223, before both the GC hook and the slot logic, so the collector's up-to-40-worktree probes cannot claim a slot or recurse
Harness on the rebased tree	tests/build-slots/run.sh serialize fifo kill nested → all scenarios passed, rc=0, wrapper stderr clean

The prior thread on the noclock scenario is resolved and its fix (79b9eb1) is carried through unchanged.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}