diff --git a/.claude/skills/cpu-reduction/SKILL.md b/.claude/skills/cpu-reduction/SKILL.md index 03d8c1d99..779f5ae87 100644 --- a/.claude/skills/cpu-reduction/SKILL.md +++ b/.claude/skills/cpu-reduction/SKILL.md @@ -37,7 +37,7 @@ df -h ~/.cache # disk — worktree target/ dirs live ps -Ao pid,ppid,pgid,%cpu,%mem,rss,etime,state,comm -r | head -40 ``` -Load well above `hw.ncpu` with nothing building → §3. A build stuck on a lock, or `~/.cache` near full → §2. +Load well above `hw.ncpu` with nothing building → §3. A build stuck on a lock, or `~/.cache` near full → §2. Start with `make build-status`: it prints which builds hold the machine-wide compile slots, who is queued behind them and for how long, token occupancy, and any build running outside the cap — a build sitting at `Compiling` with no CPU is queued behind the governor, not hung. ## 2. Old Rust builds — locks, orphans, disk diff --git a/.claude/skills/dev-loop/SKILL.md b/.claude/skills/dev-loop/SKILL.md index 55e49cd28..5f205ec8a 100644 --- a/.claude/skills/dev-loop/SKILL.md +++ b/.claude/skills/dev-loop/SKILL.md @@ -62,8 +62,7 @@ There is **no `nub build` command**. **Build politeness — the maintainer works on this machine.** -- **Job cap (already set, machine-wide):** `~/.cargo/config.toml` pins `[build] jobs = 6` of 8 perf cores. CI is unaffected. Leave it in place. -- **Background QoS — wrap every agent build:** `taskpolicy -b cargo build -p nub-cli --profile fast` (macOS background QoS → E-cores, yields to interactive) or `nice -n 10 cargo build …`. For a build already hammering the host: `renice 20 -p ` + `taskpolicy -b -p ` on the running `cargo`/`rustc` tree. +- **The machine-wide governor does the clamping, not you.** `~/.cargo/config.toml` binds `scripts/rustc-qos.sh` as the rustc wrapper (`make qos-global`; `install-dev` re-runs it), which runs every rustc at utility QoS, lets at most two builds compile at once (the rest queue first-come-first-served), and caps the host at six rustc. So a build that sits at `Compiling` with no CPU is queued, not hung: `make build-status` shows who holds the slots and who is waiting. Do not wrap builds in `taskpolicy`/`nice` yourself, and never set `NUB_BUILD_FG=1` or blank `RUSTC_WRAPPER` from an agent — the `rust-build-hygiene` skill has the rules. `~/.cargo/config.toml` also pins `[build] jobs = 6`; CI is unaffected. Leave both in place. **Why `fast`, never `release`, for iteration** (measured, macOS arm64): diff --git a/.claude/skills/rust-build-hygiene/SKILL.md b/.claude/skills/rust-build-hygiene/SKILL.md index c7fd7b4b8..f4a14e10f 100644 --- a/.claude/skills/rust-build-hygiene/SKILL.md +++ b/.claude/skills/rust-build-hygiene/SKILL.md @@ -48,9 +48,9 @@ Never fake-wait on a build with a detached shell + a `sleep`/poll loop. **A per-build cap cannot bound N builds.** Every build can be individually blameless — `--profile fast`, QoS-clamped, `jobs = 6` — and the machine still dies, because the caps multiply instead of adding. Measured 2026-08-19: 13 concurrent agent builds, every one of them already on `--profile fast`, produced a 78-way oversubscription of 10 cores — load 464, 0% idle, 36% sys, and a `reqwest` compile that normally takes ~30s taking 28 minutes. Nothing was misconfigured. There was simply no cap on the SUM. -- **`make qos-global` installs the global governor and is what actually bounds the fleet.** It registers `scripts/rustc-qos.sh` as the machine-wide rustc wrapper, where it does two jobs: clamp QoS, and hold one of `ncpu` tokens for the life of each rustc. Every build on the host shares that one pool, so one build gets the whole machine and thirteen share it. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script. -- **Never blank `RUSTC_WRAPPER`.** That is cargo's documented "no wrapper", and it opts the build out of the global cap. `scripts/rust-build.sh` used to do exactly this, which is why 10 of those 13 builds were ungoverned. `NUB_BUILD_FG=1` is the supported opt-out for a latency-sensitive foreground build. -- **`make build-status` answers "why is this machine saturated?"** It prints the sum no single session can see: load, live builds, semaphore occupancy, and which builds are outside the cap. Run it before concluding your own build is slow — it usually is not your build. +- **`make qos-global` installs the global governor and is what actually bounds the fleet.** It registers `scripts/rustc-qos.sh` as the machine-wide rustc wrapper, where it does three jobs: clamp QoS; **let at most TWO builds compile at a time** (`NUB_BUILD_SLOTS`, default 2 — every other build's first rustc waits in a first-come-first-served queue until a holder's cargo exits, dies, or goes idle for `NUB_BUILD_IDLE`=120s, so a `cargo test` running its tests or a cargo blocked on a target lock does not hold the machine); and across the compiling builds, hold one of `NUB_RUSTC_LIMIT` (default 6) tokens for the life of each rustc. Two builds over six tokens bounds the memory peak to ~6 big-crate compiles (~12 GiB) and keeps a second build's worth of cores busy; strict one-at-a-time was the first cut (2026-08-28) and was measured idling nine cores behind one starved compile while eight builds queued 25 minutes. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script. A build that queues is not stuck — after 20s it prints one `rustc-qos: this build is queued …` line on its cargo's stderr, and `make build-status` shows the holders and the queue. `rust-analyzer` is exempt so the editor never waits behind agent builds. +- **Never blank `RUSTC_WRAPPER`, and never set `NUB_BUILD_FG=1` from an agent.** Blanking is cargo's documented "no wrapper", and it opts the build out of the global cap; `scripts/rust-build.sh` used to do exactly this, which is why 10 of those 13 builds were ungoverned. `NUB_BUILD_FG=1` opts a build out of the QoS clamp and the build-slot queue (a bare `cargo` still takes rustc tokens; through `rust-build.sh` it blanks both wrapper keys, so out of the tokens too) — it exists for a HUMAN at a terminal whose build must not wait behind the fleet, and an agent that sets it recreates the 2026-08-19 incident. `NUB_BUILD_SLOTS=0` disables only the slot layer and `NUB_BUILD_SLOTS=1` restores strict one-at-a-time — both are PER-PROCESS environment knobs, read by the wrapper of the cargo that inherits them, not host-wide settings. The host-wide switch is `make build-slots-off` / `build-slots-on` (a file every wrapper checks each second, so it also releases builds already queued); it is for an emergency, and it leaves the QoS clamp and the tokens in place. +- **`make build-status` answers "why is this machine saturated?" and "why is my build not starting?"** It prints the sum no single session can see: load, which builds hold the compile slots and who is queued behind them (each tagged with its worktree), token occupancy, a STALE WRAPPER line when an older checkout's `make install-dev` downgraded the governor, and which builds are outside the cap. Run it before concluding your own build is slow or hung — a build whose rustc sits at `Compiling` for minutes with no CPU is queued, not broken. A foreground Bash call whose cargo goes silent at `Compiling` is the same thing: relaunching it puts the new cargo at the BACK of the queue. ## Performance — reuse the cache, clamp the QoS, cap the jobs diff --git a/.claude/skills/rust-build/SKILL.md b/.claude/skills/rust-build/SKILL.md index a13c8a07f..b063ae372 100644 --- a/.claude/skills/rust-build/SKILL.md +++ b/.claude/skills/rust-build/SKILL.md @@ -35,7 +35,7 @@ It prints which target dir it chose and why, then execs `cargo` with `CARGO_TARG - **QoS clamp (darwin only):** cargo runs under `taskpolicy -c utility`, so interactive work preempts fleet builds; an uncontended build still gets all cores. `NUB_BUILD_FG=1` opts out. - **Job cap on big hosts (>8 cores):** `CARGO_BUILD_JOBS = ncpu-4` unless the caller already chose (pre-set `CARGO_BUILD_JOBS`, `NUB_BUILD_JOBS`, or an explicit `-j`/`--jobs` — cargo's flag outranks the env var). -These cover builds going THROUGH the wrapper (or make). Direct `cargo` invocations are clamped by a machine-global control: `make qos-global` installs `scripts/rustc-qos.sh` as the cargo `rustc-wrapper` in `~/.cargo/config.toml`, so every rustc on the host compiles at utility QoS (`install-dev` re-runs it, so it self-heals). Same `NUB_BUILD_FG=1` opt-out; toggling it does not invalidate fingerprints. +These cover builds going THROUGH the wrapper (or make). Direct `cargo` invocations are clamped by a machine-global control: `make qos-global` installs `scripts/rustc-qos.sh` as the cargo `rustc-wrapper` in `~/.cargo/config.toml`, so every rustc on the host compiles at utility QoS, at most TWO builds compile at a time (the rest queue first-come-first-served; `make build-status` shows the holders and the queue), and the compiling builds share `NUB_RUSTC_LIMIT` (default 6) rustc tokens (`install-dev` re-runs it, so it self-heals). `NUB_BUILD_FG=1` opts a HUMAN's foreground build out of the clamp and the queue (and, through this wrapper, out of the tokens too) — never set it from an agent; `NUB_BUILD_SLOTS=0` in a build's environment disables only the slot layer for that build. Toggling any of these does not invalidate fingerprints. Details: the `rust-build-hygiene` skill. ## Why one shared target dir diff --git a/.githooks/pre-push b/.githooks/pre-push index 52997ae7c..0274d05a4 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -213,7 +213,7 @@ fi # and runs only when the pushed range touches tests/cross-runtime/**. Like the # site gate, an undecidable range runs the check rather than skipping it. # -# The push refs on stdin are read ONCE here and replayed to the site gate below. +# The push refs on stdin are read ONCE here and replayed to every gate below. # Bypass: NUB_SKIP_README_TABLE_CHECK=1 git push · git push --no-verify PUSH_REFS=$(cat) if [ -z "${NUB_SKIP_README_TABLE_CHECK:-}" ] && [ -n "$SKILLS_ROOT" ] && [ -f "$SKILLS_ROOT/tests/cross-runtime/readme-table.mjs" ]; then @@ -250,6 +250,25 @@ PUSHREFS fi fi +# A change to the machine-global rustc governor must bump its version stamp, or +# build-status can no longer tell a stale installed copy from a current one. +# The base is the MERGE-BASE with trunk, not the branch's previous push — the +# stamp names a version of the protocol, so CI's build-slots job wants one bump +# per pull request and this gate must want the same, not one per push. +# Bypass: NUB_SKIP_QOS_VERSION_CHECK=1 git push +# +if [ -z "${NUB_SKIP_QOS_VERSION_CHECK:-}" ] && [ -n "$SKILLS_ROOT" ] \ + && [ -x "$SKILLS_ROOT/scripts/rustc-qos-version-check.sh" ]; then + while read -r _local_ref local_sha _remote_ref remote_sha; do + case "$local_sha" in *[!0]*) ;; *) continue ;; esac + base=$(git merge-base "$local_sha" origin/main 2>/dev/null) || base="" + [ -n "$base" ] || continue + (cd "$SKILLS_ROOT" && scripts/rustc-qos-version-check.sh "$base" "$local_sha") || exit 1 + done </dev/null || echo 4 ) sem=${NUB_RUSTC_SEM_DIR:-$HOME/.cache/nub/rustc-sem} -limit=${NUB_RUSTC_LIMIT:-$ncpu} +limit=${NUB_RUSTC_LIMIT:-6} +bdir=${NUB_BUILD_SEM_DIR:-$HOME/.cache/nub/build-sem} +bslots=${NUB_BUILD_SLOTS:-2} +cfg="$HOME/.cargo/config.toml" cargos=$(ps -Ao command= | grep -c '[b]in/cargo') rustcs=$(ps -Ao command= | grep -c '[b]in/rustc') -wrappers=$(ps -Ao command= | grep -c '[r]ustc-qos.sh') -held=$(find "$sem" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l | tr -d ' ') +# The wrapper is installed under two names (see qos-global.sh); count both. +wrappers=$(ps -Ao command= | grep -c -e '[r]ustc-qos.sh' -e '[r]ustc-gov.sh') +held=$(find "$sem" -mindepth 1 -maxdepth 1 -type d -name '[0-9]*' 2>/dev/null | wc -l | tr -d ' ') runnable=$(ps -Ao state= | cut -c1 | grep -c R) +# The env of a live cargo, one `KEY=value` per line, for the wrapper-key and +# worktree tells below; `tree` is the last path component of its PWD, which is +# what tells identical `cargo test -p nub-cli` rows apart. +envof() { ps eww -o command= -p "$1" 2>/dev/null | tr ' ' '\n'; } +tree() { envof "$1" | sed -n 's|^PWD=.*/||p' | head -1; } printf '\n== host ==\n' printf ' cores %s %s\n' "$ncpu" "$(uptime | sed 's/.*load averages*://')" printf ' runnable threads %s rustc %s cargo %s\n' "$runnable" "$rustcs" "$cargos" printf ' disk %s\n' "$(df -h /System/Volumes/Data 2>/dev/null | awk 'NR==2{print $4" free ("$5" used)"}')" +printf '\n== build slots (at most %s builds compile at once; the rest queue) ==\n' "$bslots" +if ! grep -q 'rustc-wrapper = ' "$cfg" 2>/dev/null; then + printf ' NOT ACTIVE — run: make qos-global\n' +elif [ -e "$bdir/off" ]; then + printf ' OFF SWITCH SET (%s): every build compiles unqueued — run: make build-slots-on\n' "$bdir/off" +else + now=$(date +%s) + for d in "$bdir"/slot/*/; do + [ -d "$d" ] || continue + p=$(cat "$d/pid" 2>/dev/null) + st=$(cat "$d/stamp" 2>/dev/null) + if [ -z "$st" ]; then + printf ' slot %s NO STAMP (a claim in progress, or the corpse of one; retired once a minute old)\n' "$(basename "$d")" + continue + fi + if [ -n "$p" ] && ! kill -0 "$p" 2>/dev/null; then + # Reclaim is lazy — the next wrapper to poll retires this — so a dead + # holder on a quiet host is expected, not a leak. + printf ' slot %s cargo %-7s DEAD (retired by the next build to poll)\n' "$(basename "$d")" "$p" + continue + fi + live=0 + for m in "$d"active/*; do [ -e "$m" ] && kill -0 "${m##*/}" 2>/dev/null && live=$((live + 1)); done + printf ' slot %s cargo %-7s %8s %s compiling, last start %ss ago %s [%s]\n' \ + "$(basename "$d")" "$p" "$(ps -o etime= -p "$p" 2>/dev/null | tr -d ' ')" "$live" "$((now - st))" \ + "$(ps -o command= -p "$p" 2>/dev/null | sed 's|^[^ ]*/bin/||' | cut -c1-60)" "$(tree "$p")" + done + n=0 + for t in "$bdir"/queue/*; do + [ -e "$t" ] || continue + q=${t##*/} + case $q in *.*) continue ;; esac + # Read once: the owner truncates-then-writes and finally removes this file, + # so it can be empty or gone between the guard above and here. + first=""; { read -r first < "$t"; } 2>/dev/null || first="" + [ -n "$first" ] || continue + n=$((n + 1)) + printf ' queued cargo %-7s waiting %4ss %s [%s]\n' "$q" "$((now - first))" \ + "$(ps -o command= -p "$q" 2>/dev/null | sed 's|^[^ ]*/bin/||' | cut -c1-60)" "$(tree "$q")" + done + printf ' %s/%s slots in use, %s queued\n' \ + "$(find "$bdir/slot" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l | tr -d ' ')" "$bslots" "$n" +fi + +# The layer holds only for builds that inherited THIS wrapper. Every stale +# checkout's qos-global.sh copies its own older rustc-qos.sh over the installed +# one on `make install-dev`, so compare versions here rather than assume. +want=$(sed -n 's/^# rustc-qos-version: \([0-9]*\).*/\1/p' "$(dirname "$0")/rustc-qos.sh" 2>/dev/null | head -1) +have=$(sed -n 's/^# rustc-qos-version: \([0-9]*\).*/\1/p' "$HOME/.cargo/rustc-qos.sh" 2>/dev/null | head -1) +if [ -n "$want" ] && [ "${have:-0}" -lt "$want" ] 2>/dev/null; then + printf ' STALE WRAPPER: ~/.cargo/rustc-qos.sh is v%s, this tree ships v%s — builds since the downgrade run an older protocol; run: make qos-global\n' "${have:-0}" "$want" +fi + printf '\n== global rustc semaphore ==\n' if [ -d "$sem" ]; then printf ' %s/%s tokens held %s wrapper shells (held + waiting)\n' "$held" "$limit" "$wrappers" @@ -36,22 +101,37 @@ else fi printf '\n== live cargo invocations ==\n' -ps -Ao pid=,etime=,command= | grep '[b]in/cargo' | while read -r pid etime rest; do - # A build is governed only if it inherited the machine-global wrapper. An - # explicitly EMPTY RUSTC_WRAPPER is cargo's documented "no wrapper", so that is - # the tell for an escapee -- not an ABSENT variable, which means "inherit". - w=$(ps eww -o command= -p "$pid" 2>/dev/null | tr ' ' '\n' \ - | grep '^RUSTC_WRAPPER=' | head -1) - # An explicitly EMPTY RUSTC_WRAPPER is cargo's documented "no wrapper", so such - # a build is outside the full semaphore -- but NOT ungoverned: qos-global also - # registers the wrapper as rustc-workspace-wrapper, which those callers do not - # blank, so workspace crates still take tokens. Deps and vendor/aube do not. +partial=0 +for pid in $(ps -Ao pid=,command= | awk '/[b]in\/cargo/ {print $1}'); do + etime=$(ps -o etime= -p "$pid" 2>/dev/null | tr -d ' ') + rest=$(ps -o command= -p "$pid" 2>/dev/null) + [ -n "$rest" ] || continue + env=$(envof "$pid") + w=$(printf '%s\n' "$env" | grep '^RUSTC_WRAPPER=' | head -1) + ww=$(printf '%s\n' "$env" | grep -x -e 'NUB_BUILD_FG=1' -e 'RUSTC_WORKSPACE_WRAPPER=' | head -1) + # An explicitly EMPTY RUSTC_WRAPPER is cargo's documented "no wrapper" (an + # ABSENT variable means "inherit"), so such a build is outside the full + # semaphore -- but NOT ungoverned: qos-global also registers the wrapper as + # rustc-workspace-wrapper, so workspace crates still take tokens while deps + # and vendor/aube do not. No in-tree caller produces this shape any more + # (rust-build.sh blanks both keys or neither); it is the mark of a STALE + # checkout's rust-build.sh, or a hand-rolled config. case "$w" in - 'RUSTC_WRAPPER=') mark='partial (workspace)' ;; + 'RUSTC_WRAPPER=') mark='partial (workspace)'; partial=$((partial + 1)) ;; *) mark='governed' ;; esac - printf ' %-7s %8s %-19s %s\n' "$pid" "$etime" "$mark" \ - "$(printf '%s' "$rest" | sed 's|^[^ ]*/bin/||' | cut -c1-72)" -done 2>/dev/null + # A build the human marked foreground skips the slot queue by design; say + # so rather than let it read as an escapee. (A build holding neither slot + # nor ticket is otherwise NOT suspect — that is what idle reclaim looks like.) + # rust-build.sh unsets NUB_BUILD_FG before its exec, so for that path the + # tell is BOTH wrapper keys blank; the env var still catches a foreground + # `make build` or a bare `cargo build`. + if [ "$ww" = 'NUB_BUILD_FG=1' ] || { [ "$w" = 'RUSTC_WRAPPER=' ] && [ -n "$ww" ]; }; then + mark='foreground (FG)' + fi + printf ' %-7s %8s %-19s %s [%s]\n' "$pid" "$etime" "$mark" \ + "$(printf '%s' "$rest" | sed 's|^[^ ]*/bin/||' | cut -c1-64)" "$(tree "$pid")" +done +[ "$partial" -gt 0 ] && printf ' %s build(s) partial: their dependency crates compile outside the cap\n' "$partial" printf '\n' diff --git a/scripts/qos-global.sh b/scripts/qos-global.sh index 41e10862c..6c5db5fb6 100755 --- a/scripts/qos-global.sh +++ b/scripts/qos-global.sh @@ -5,36 +5,37 @@ # darwin-only; refuses to clobber a foreign wrapper (e.g. sccache). # `make install-dev` depends on this, so the installed copy self-heals. # -# BOTH KEYS ARE INSTALLED, and the second one is not redundant. `rust-build.sh` -# historically blanked RUSTC_WRAPPER, and 58 of 61 live worktrees still carry -# that version (measured 2026-08-19) — every build launched from one opts itself -# out of `rustc-wrapper` and out of the global concurrency semaphore with it. -# Those scripts do NOT blank RUSTC_WORKSPACE_WRAPPER, so registering the same -# wrapper there keeps the workspace crates governed no matter how stale the -# checkout is. Coverage is partial by construction (workspace members only, so -# not vendor/aube or crates.io deps) — it is a floor under stale checkouts, not -# a replacement for the wrapper proper. The wrapper is re-entrant, so a workspace -# crate running through both hops still takes exactly one token. +# BOTH KEYS ARE INSTALLED, and the second one is not redundant — it is the floor +# under STALE CHECKOUTS, which come in two shapes: +# - `rust-build.sh` historically blanked RUSTC_WRAPPER, and 58 of 61 live +# worktrees still carried that version on 2026-08-19. Such a build opts +# itself out of `rustc-wrapper`, but not out of RUSTC_WORKSPACE_WRAPPER, so +# binding the same wrapper there keeps its workspace crates governed +# (partial by construction: not vendor/aube or crates.io deps). +# - Every stale checkout's qos-global.sh copies its own older rustc-qos.sh +# over ~/.cargo/rustc-qos.sh on `make install-dev`, silently downgrading the +# machine-wide governor. Checkouts older than this second name leave +# ~/.cargo/rustc-gov.sh alone, so a governor survives under that name; +# newer stale checkouts clobber both, which `make build-status` reports as +# STALE WRAPPER. The wrapper's body is one compound command precisely so a +# copy over the live file cannot truncate a wrapper mid-compile. +# The wrapper is re-entrant, so a workspace crate running through both hops +# still takes exactly one token. set -eu [ "$(uname)" = "Darwin" ] || { echo "qos-global: darwin-only, skipping"; exit 0; } dir=$(cd "$(dirname "$0")" && pwd) cfg="$HOME/.cargo/config.toml" wrapper="$HOME/.cargo/rustc-qos.sh" -# SECOND NAME, DELIBERATELY. Every stale checkout carries its own qos-global.sh -# whose FIRST act is an unconditional `cp` over ~/.cargo/rustc-qos.sh, and -# `make install-dev` calls it. So one `make install-dev` from any of the 58 -# worktrees still on the old script silently downgrades the machine-wide wrapper -# back to a QoS-only clamp and the concurrency semaphore just disappears -- no -# error, no output, load climbs again an hour later. Those scripts have never -# heard of this path, so installing the same wrapper under a second name and -# binding THAT to rustc-workspace-wrapper keeps a governor alive through the -# clobber. Both names are the same file, and the wrapper is re-entrant, so a -# crate that runs through both hops still takes exactly one token. -governor="$HOME/.cargo/rustc-gov.sh" +governor="$HOME/.cargo/rustc-gov.sh" # the second name, see above mkdir -p "$HOME/.cargo" -cp "$dir/rustc-qos.sh" "$wrapper" -cp "$dir/rustc-qos.sh" "$governor" -chmod +x "$wrapper" "$governor" +# Write-then-rename, never `cp` over the live file: sh reads a script +# incrementally, so every wrapper mid-compile machine-wide would read the new +# bytes at its old offset. A rename gives them their old inode to finish on. +for dst in "$wrapper" "$governor"; do + cp "$dir/rustc-qos.sh" "$dst.tmp.$$" + chmod +x "$dst.tmp.$$" + mv "$dst.tmp.$$" "$dst" +done # Refuse to fight a foreign wrapper (sccache and friends own the same slot). if [ -f "$cfg" ] && grep -q '^[[:space:]]*rustc-wrapper' "$cfg" \ diff --git a/scripts/rust-build.sh b/scripts/rust-build.sh index 601deef50..223716aeb 100755 --- a/scripts/rust-build.sh +++ b/scripts/rust-build.sh @@ -287,20 +287,24 @@ if [ "${NUB_BUILD_FG:-}" != "1" ] && [ "$(uname)" = "Darwin" ] \ qos="taskpolicy -c utility" fi -# The machine-global rustc wrapper (make qos-global) now carries the GLOBAL -# CONCURRENCY SEMAPHORE, not just a QoS clamp, so this invocation must let it -# run. It previously blanked RUSTC_WRAPPER — sound when the wrapper only +# The machine-global rustc wrapper (make qos-global) carries the BUILD SLOTS and +# the GLOBAL CONCURRENCY SEMAPHORE, not just a QoS clamp, so this invocation +# must let it run. It previously blanked RUSTC_WRAPPER — sound when the wrapper only # re-applied a clamp cargo had already applied, and the direct cause of the # 2026-08-19 saturation once the semaphore moved there: 10 of the 13 concurrent # builds went through this script and every one of them opted itself out of the # only machine-wide cap that existed. The per-cargo CARGO_BUILD_JOBS above stays # — it is blind to sibling builds, so it bounds ONE build, never the fleet. # -# NUB_BUILD_FG=1 still opts fully out of both, which is the documented meaning of -# a latency-sensitive foreground build; it is passed by blanking RUSTC_WRAPPER -# rather than by exporting the var, so the unset below keeps holding. +# NUB_BUILD_FG=1 still opts fully out, which is the documented meaning of a +# HUMAN's latency-sensitive foreground build; it is passed by blanking BOTH +# wrapper keys rather than by exporting the var, so the unset below keeps +# holding. Both, because qos-global binds the same governor to +# rustc-workspace-wrapper as a floor under stale checkouts, and a foreground +# build whose workspace crates still queued behind the fleet would be the +# opposite of what was asked for. wrapper_off="" -[ "${NUB_BUILD_FG:-}" = "1" ] && wrapper_off="RUSTC_WRAPPER=" +[ "${NUB_BUILD_FG:-}" = "1" ] && wrapper_off="RUSTC_WRAPPER= RUSTC_WORKSPACE_WRAPPER=" printf 'rust-build: %s\n CARGO_TARGET_DIR=%s jobs=%s qos=%s sem=%s\n' \ "$why" "$target" "${CARGO_BUILD_JOBS:-default}" "${qos:-none}" \ diff --git a/scripts/rustc-qos-version-check.sh b/scripts/rustc-qos-version-check.sh new file mode 100755 index 000000000..5ed4361f7 --- /dev/null +++ b/scripts/rustc-qos-version-check.sh @@ -0,0 +1,30 @@ +#!/bin/sh +# rustc-qos-version-check — refuse a change to scripts/rustc-qos.sh that leaves +# its `# rustc-qos-version: N` stamp where it was. +# +# build-status compares the INSTALLED ~/.cargo/rustc-qos.sh against the tree's +# copy by that stamp to report a stale governor. Nothing else bumps it, so the +# first behavior change that forgets would make every stale host read as +# current — the exact failure the stamp exists to catch. One bump per pull +# request: the pre-push hook runs it against the merge-base with trunk, CI +# against the pull request's base. +# +# scripts/rustc-qos-version-check.sh [] +set -eu +base=$1 +head=${2:-HEAD} +f=scripts/rustc-qos.sh +# 0 untouched, 1 changed, anything else is git failing (a bad rev) — which must +# not read as "untouched". +# shellcheck disable=SC2015 # `exit` never returns, so the `||` arm is the changed case only +git diff --quiet "$base" "$head" -- "$f" && exit 0 || [ $? -eq 1 ] +stamp() { git show "$1:$f" 2>/dev/null | sed -n 's/^# rustc-qos-version: \([0-9]*\).*/\1/p' | head -1; } +old=$(stamp "$base"); new=$(stamp "$head") +if [ -z "$new" ]; then + echo "rustc-qos-version-check: $f has no '# rustc-qos-version: N' line" >&2; exit 1 +fi +if [ "${old:-0}" -ge "$new" ] 2>/dev/null; then + echo "rustc-qos-version-check: $f changed but its stamp is still v$new (base v${old:-none}) — bump '# rustc-qos-version:' so build-status can tell a stale install from a current one" >&2 + exit 1 +fi +exit 0 diff --git a/scripts/rustc-qos.sh b/scripts/rustc-qos.sh index 9088206fb..c1f3dc417 100755 --- a/scripts/rustc-qos.sh +++ b/scripts/rustc-qos.sh @@ -1,14 +1,32 @@ #!/bin/sh -# rustc-qos — machine-global cargo rustc-wrapper. Two jobs, both about stopping a +# rustc-qos-version: 11 (build-status compares the installed copy against this) +# rustc-qos — machine-global cargo rustc-wrapper. Three jobs, all about stopping a # fleet of concurrent agent builds from bricking a 10-core dev host: # # 1. QoS clamp (darwin): every rustc runs at 'utility', so builds always yield # to interactive work. -# 2. GLOBAL CONCURRENCY SEMAPHORE: at most N rustc run machine-wide, across -# every worktree, every cargo, and every entry point. +# 2. BUILD SLOTS: at most NUB_BUILD_SLOTS cargo invocations (default 2) may be +# COMPILING at once, machine-wide, served first-come first-served. Every +# other build's rustc waits at its first compile until a slot frees. +# 3. GLOBAL RUSTC SEMAPHORE: within the builds that hold a slot, at most +# NUB_RUSTC_LIMIT rustc (default 6) run machine-wide, across every worktree, +# every cargo, every entry point. # -# WHY THE SEMAPHORE LIVES HERE AND NOT IN CARGO'S JOB COUNT. Every per-build cap -# is blind to its siblings, so each cargo keeps its promise individually and the +# WHY A BUILD-LEVEL CAP ON TOP OF THE RUSTC-LEVEL ONE. The rustc semaphore bounds +# how many rustc PROCESSES exist, and nothing else. Each holds a jobserver from +# its own cargo, so ten governed rustc still run up to `jobs` LLVM threads apiece +# (measured 2026-08-28: 10 rustc x 7 threads on 10 cores), and ten concurrent +# compiles of the big crates peak at 1-3 GiB each on a host already carrying +# ~40 GiB of editors, browsers and agent sessions — that is the swap storm the +# maintainer asked to end. Capping BUILDS bounds who competes: two builds share +# the six tokens (~12 GiB at the big crates' peak) and the rest wait in a queue +# instead of thrashing alongside them, so the builds that run finish at speed. +# Strict one-at-a-time was the first cut and idled nine cores behind a single +# starved compile while eight builds queued for 25 minutes; two slots over six +# tokens is the maintainer's pick (2026-08-28). +# +# WHY THE CAPS LIVE HERE AND NOT IN CARGO'S JOB COUNT. Every per-build cap is +# blind to its siblings, so each cargo keeps its promise individually and the # machine pays collectively: 13 concurrent agent builds x `jobs = 6` is a 78-way # oversubscription of 10 cores (measured 2026-08-19 — load 464, 0% idle, 36% sys, # 230 runnable threads, every one of those builds already on --profile fast). A @@ -18,15 +36,59 @@ # what makes it hold, since the measured failure was precisely that 3 of the 13 # builds had bypassed scripts/rust-build.sh and its per-build clamp entirely. # -# WHY BLOCKING CANNOT DEADLOCK. Cargo spawns rustc only for units whose -# dependencies are already built, so any two concurrent rustc are independent by -# construction and one waiting can never block another's completion. A token is -# held for the life of ONE rustc — seconds to a couple of minutes — so a wait is -# bounded by a single compile, never by a whole build. +# WHY BLOCKING CANNOT DEADLOCK. A slot belongs to the OUTERMOST cargo in this +# rustc's ancestry, so a nested cargo (a build script that runs cargo, `make` in +# front of cargo) inherits its parent's slot rather than queueing behind it. A +# slot-holding build waits on nothing but rustc tokens, each held for the life +# of ONE rustc, so a queued build's wait is bounded by the builds ahead of it. +# Within a build, cargo spawns rustc only for units whose dependencies are +# already built, so any two concurrent rustc are independent by construction and +# one waiting can never block another's completion. +# +# WHY A SLOT CAN BE RECLAIMED FROM A LIVE CARGO. `cargo test` keeps running long +# after its last compile, and a cargo blocked on another target dir's lock never +# compiles at all. Holding the slot through either would stall every other build +# for nothing, so a slot is reclaimable once NO compile of its build is alive and +# none has started for NUB_BUILD_IDLE seconds. A build that then compiles again +# re-queues AT ITS ORIGINAL PLACE — its ticket carries the time it first queued +# — so losing the slot costs it one other build's turn, never a trip to the back +# of the line (measured 2026-08-28: a build that lost its slot in a build-script +# gap under load 40 re-queued behind four others for its last crate). Build +# scripts (cmake for zstd, ring's C) are the compile-adjacent gap that exceeds a +# short window, hence the 120s default. +# +# TWO LIVELOCK SHAPES, AND WHY NEITHER HOLDS. A queue ticket whose waiters all +# died while its cargo lives on would sit at the head forever, so every waiter +# heartbeats its ticket each loop and a ticket not refreshed for 15s is ignored +# and pruned. A live-compile marker whose pid was recycled by an unrelated +# long-lived process would pin its slot against idle reclaim, so a marker also +# records when it was written and one older than NUB_BUILD_MAXCOMPILE (30 min) +# no longer counts as live — no single compile here runs that long. +# +# ONE ACCEPTED RACE. A reaper that scans a holder's markers (none live), then +# the holder's next compile marks the slot and proceeds, then the reaper +# retires it: that compile runs unslotted while another build claims the slot. +# Bounded to one compile, and the state converges (the holder re-queues at its +# place). Closing it would need a mutex shared by idle-retire and every +# compile start; the cap breach it allows is smaller than that mutex's cost. # # FAIL-OPEN ON EVERY PATH. This sits in front of every rustc on the machine, so a -# bug here breaks every build in every worktree. Unwritable cache dir, a bad -# limit, exhausted retries: each falls through to running rustc unthrottled. +# bug here breaks every build in every worktree. No cargo ancestor, a state dir +# that is missing or unwritable (at entry or mid-wait — a full disk, an operator's +# `rm -rf`), a cargo that died while its rustc queued, a bad tunable, exhausted +# retries: each falls through to running rustc unthrottled. A bad tunable takes +# its default, except NUB_BUILD_SLOTS=0 / NUB_RUSTC_LIMIT=0, which switch that +# layer off for the build. rust-analyzer's `cargo check` is exempt from the build +# slots by design (it stays under the rustc semaphore) so the editor never queues +# behind the agent fleet. `touch $NUB_BUILD_SEM_DIR/off` is the host-wide off +# switch: every wrapper checks it on entry and on every wait iteration, so it +# releases builds already queued (`make build-slots-off` / `build-slots-on`). +# +# THE BODY IS ONE COMPOUND COMMAND. sh reads a script incrementally, so an +# installer that copies over the live file (this repo's own did, until 2026-08-28, +# and every checkout that predates that still does) ends each running wrapper at +# the new EOF with exit 0 — cargo then records a unit as built that no rustc ever +# produced. Braced, the whole body is parsed before any of it runs. # # Installed by `make qos-global` into ~/.cargo (config.toml rustc-wrapper -> a # stable copy at ~/.cargo/rustc-qos.sh). Deliberately NOT a tracked @@ -34,10 +96,21 @@ # contributors), and machine-global also covers stale worktrees and file:// # clones that predate any commit. Toggling the wrapper does not invalidate cargo # fingerprints (verified 2026-07-24), so wrapped and unwrapped builds share a -# target dir without rebuild churn. NUB_BUILD_FG=1 opts out of the QoS clamp. +# target dir without rebuild churn. NUB_BUILD_FG=1 is the HUMAN's foreground +# escape: it skips the QoS clamp and the build-slot queue (the rustc tokens still +# apply). It is for a person at a terminal, never for an agent build — that is +# how the 2026-08-19 oversubscription came about. `make build-status` shows the +# slots, the queue and the token pool; a build queued for 20s says so once on +# its cargo's stderr, so a silent `Compiling` line is never mistaken for a hang. # -# Tunables (all optional): NUB_RUSTC_LIMIT (concurrent rustc, default = ncpu), -# NUB_RUSTC_SEM_DIR (token dir), NUB_RUSTC_SEM_TRIES (retry ceiling). +# Tunables (all optional, read per process): NUB_BUILD_SLOTS (concurrent +# compiling builds, default 2; 0 disables the layer), NUB_BUILD_IDLE (seconds, +# default 120), NUB_BUILD_WAIT (queue ceiling in seconds, default 3600, then fail +# open), NUB_BUILD_MAXCOMPILE (seconds a compile marker stays live, default +# 1800), NUB_BUILD_NOCLOCK_MAX (consecutive clock misses before failing open, +# default 60), NUB_BUILD_SEM_DIR; NUB_RUSTC_LIMIT (concurrent rustc, default 6), +# NUB_RUSTC_SEM_DIR, NUB_RUSTC_SEM_TRIES (retry ceiling, default 1500 x 0.4s). +{ # Cargo execs this wrapper for capability probes (`rustc -vV`, `--print …`) at # startup and during build-script target detection. Those return in milliseconds @@ -49,38 +122,392 @@ for _a in "$@"; do done _qos="" -if [ "${NUB_BUILD_FG:-}" != "1" ] && [ "$(uname)" = "Darwin" ] \ +if [ "${NUB_BUILD_FG:-}" != "1" ] && [ "$(uname 2>/dev/null)" = "Darwin" ] \ && command -v taskpolicy >/dev/null 2>&1; then _qos="taskpolicy -c utility" fi # RE-ENTRANCY. This script is installed as BOTH rustc-wrapper and # rustc-workspace-wrapper, and cargo composes them for a workspace crate -- -# outer wrapper, then inner. Without this guard each such rustc would take TWO -# tokens, and a pool fully held by outer wrappers all waiting on inner ones is a -# genuine deadlock. The marker is set only when a token is actually HELD, so an -# outer that failed open still lets the inner try. +# outer wrapper, then inner. The outer settles everything (slot, token, or the +# decision to fail open) and marks it, so the inner just execs; without the +# guard each such rustc would take TWO tokens, and a pool fully held by outer +# wrappers all waiting on inner ones is a genuine deadlock. if [ "${NUB_RUSTC_SEM_HELD:-}" = "1" ]; then exec "$@" fi -_ncpu=$( { sysctl -n hw.ncpu || nproc; } 2>/dev/null || echo 4 ) -_limit=${NUB_RUSTC_LIMIT:-$_ncpu} +_limit=${NUB_RUSTC_LIMIT:-6} +_tries_max=${NUB_RUSTC_SEM_TRIES:-1500} _sem=${NUB_RUSTC_SEM_DIR:-$HOME/.cache/nub/rustc-sem} _slot="" +# ---------------------------------------------------------------- build slots +# EVERY command substitution in this script silences its stderr: a wrapper +# SIGKILLed mid-`$(…)` leaves the child writing to a closed pipe, and where +# SIGPIPE is ignored (anything under a Node parent — the CI runner, an agent +# harness) that is a `Broken pipe` / `i/o error` line in cargo's output +# rather than a silent death. `date` and BSD `awk` have both done it. +_now() { date +%s 2>/dev/null; } +_bslots=${NUB_BUILD_SLOTS:-2} +_bidle=${NUB_BUILD_IDLE:-120} +_bwait=${NUB_BUILD_WAIT:-3600} +_bmax=${NUB_BUILD_MAXCOMPILE:-1800} +_nomax=${NUB_BUILD_NOCLOCK_MAX:-60} +# A non-numeric tunable makes every [ … -gt "$var" ] below an error, which spews +# to the wrapper's real stderr and silently disables the guard it gates — for +# NUB_BUILD_WAIT that is the fail-open valve itself. Fall back to the default. +[ "$_bslots" -ge 0 ] 2>/dev/null || _bslots=2 +[ "$_bidle" -ge 0 ] 2>/dev/null || _bidle=120 +[ "$_bwait" -ge 0 ] 2>/dev/null || _bwait=3600 +[ "$_bmax" -ge 0 ] 2>/dev/null || _bmax=1800 +[ "$_nomax" -ge 1 ] 2>/dev/null || _nomax=60 +[ "$_limit" -ge 0 ] 2>/dev/null || _limit=6 +[ "$_tries_max" -ge 0 ] 2>/dev/null || _tries_max=1500 +_bdir=${NUB_BUILD_SEM_DIR:-$HOME/.cache/nub/build-sem} +_bslot="" +_cargo="" +_exempt="" + +# Collect a state directory's dead entries. An entry is named for the pid it +# belongs to and lives exactly as long as that pid; a `name.` is a +# temp file mid-rename, garbage once its writer is gone. Nothing here is aged. +_sweep() { + for _e in "$1"/*; do + [ -e "$_e" ] || continue + _n=${_e##*/}; _n=${_n#claim.} + case $_n in + *.*) kill -0 "${_n##*.}" 2>/dev/null || rm -rf "$_e" 2>/dev/null ;; + *) kill -0 "$_n" 2>/dev/null || rm -rf "$_e" 2>/dev/null ;; + esac + done +} + +# One `ps -A` snapshot and an awk walk up from this shell: the OUTERMOST cargo is +# the build's identity (see the deadlock note above), and any rust-analyzer in +# the chain exempts the build. One fork, ~20ms, versus two forks per ancestor. +# +# The snapshot is the wrapper's one real cost (~190ms under load), and cargo +# execs this wrapper hundreds of times per build from the SAME parent, so the +# result is cached per parent pid. A recycled pid cannot alias: the entry also +# records the parent's start time and is discarded when that no longer matches. +_pcache="" +_walk="" +if [ "$_bslots" -gt 0 ] && [ ! -e "$_bdir/off" ]; then + _pcache="$_bdir/parent/$PPID" + _pstart=$(ps -o lstart= -p "$PPID" 2>/dev/null) + if [ -n "$_pstart" ] && [ -r "$_pcache" ]; then + { IFS= read -r _pcached && IFS= read -r _walk; } 2>/dev/null < "$_pcache" || _walk="" + [ "$_pcached" = "$_pstart" ] || _walk="" + fi + if [ -z "$_walk" ]; then + _walk=$(ps -Ao pid=,ppid=,comm= 2>/dev/null | awk -v start="$$" ' + { pp[$1] = $2; c = $3; for (i = 4; i <= NF; i++) c = c " " $i; comm[$1] = c } + END { + p = start; found = ""; ex = ""; n = 0 + while (p > 1 && n < 64 && (p in pp)) { + if (comm[p] ~ /(^|\/)cargo$/) found = p + if (comm[p] ~ /rust-analyzer/) ex = "ra" + p = pp[p]; n++ + } + print found, ex + }' 2>/dev/null) + if [ -n "$_pstart" ] && mkdir -p "${_pcache%/*}" 2>/dev/null; then + # A new parent is rare (once per cargo), so this is where dead entries go. + _sweep "${_pcache%/*}" + { printf '%s\n%s\n' "$_pstart" "$_walk" > "$_pcache.$$" \ + && mv "$_pcache.$$" "$_pcache"; } 2>/dev/null + fi + fi + _cargo=${_walk%% *} + _exempt=${_walk#* } + [ "$_exempt" = "$_walk" ] && _exempt="" + [ "${NUB_BUILD_FG:-}" = "1" ] && _exempt="fg" +fi + +# Retire a slot dir ATOMICALLY: rename it out of the table first, so of two +# waiters that both judge one slot reclaimable exactly one wins the rename and +# the other's rm can never wipe a slot the winner has already re-created. +_retire() { + _x=${1%/} + _tomb="$_bdir/reap/${_x##*/}.$$" + mv "$_x" "$_tomb" 2>/dev/null && rm -rf "$_tomb" 2>/dev/null +} + +# A slot's stamp and its live markers are written by rename, never in place: a +# reader that lands between a `>`'s truncate and its write sees an EMPTY file, +# and an empty stamp read as "stampless" retired a live slot (reproduced at a +# 30% torn-read rate on this host). The marker temp is dot-prefixed so the +# `active/*` scan never sees it. Both are silent on a directory that vanished. +_stamp() { { printf '%s\n' "$_tnow" > "$1/.stamp.$$" && mv "$1/.stamp.$$" "$1/stamp"; } 2>/dev/null; } +_mark() { { printf '%s\n' "$_tnow" > "$1/active/.$$" && mv "$1/active/.$$" "$1/active/$$"; } 2>/dev/null; } + +# Bring the slot table up to date: drop a slot whose cargo is gone, or whose +# build has no live compile and has not started one for NUB_BUILD_IDLE seconds; +# drop a queue ticket whose cargo is gone or whose waiters have stopped +# heartbeating. A compile is "live" while the pid in its marker is: that pid is +# the shell that waits on the rustc (a slot holder never execs), so the marker +# outlives the compile by nothing and a SIGKILL leaves only a stale pid that +# `kill -0` rejects. The marker's content is the time it was written, so a +# recycled pid stops counting after NUB_BUILD_MAXCOMPILE. Every number read from +# the table is validated first: a corrupt file must never be a fatal +# arithmetic error in the wrapper of every rustc on the machine. +_reap() { + for _d in "$_bdir"/slot/*/; do + [ -d "$_d" ] || continue + _p=""; { read -r _p < "$_d/pid"; } 2>/dev/null || _p="" + if [ -n "$_p" ] && ! kill -0 "$_p" 2>/dev/null; then + _retire "$_d"; continue # holder cargo is gone + fi + _live=0 + for _m in "$_d"active/*; do + [ -e "$_m" ] || continue + if kill -0 "${_m##*/}" 2>/dev/null; then + # An unreadable age with a live pid counts as live, never as expired: + # the fallback direction decides whether a torn read kills a compile. + _w=""; { read -r _w < "$_m"; } 2>/dev/null || _w="" + [ "$_w" -ge 0 ] 2>/dev/null || _w=$_tnow + [ $(( _tnow - _w )) -le "$_bmax" ] && { _live=1; continue; } + fi + rm -f "$_m" 2>/dev/null + done + [ "$_live" = 1 ] && continue + # No live marker. Read the stamp only now: a holder's _release refreshes + # the stamp and THEN drops its marker, so a marker seen gone implies the + # fresh stamp is already there. + _s=""; { read -r _s < "$_d/stamp"; } 2>/dev/null || _s="" + if ! [ "$_s" -ge 0 ] 2>/dev/null; then + # No stamp and no live compile: a claim in progress (it lands within + # milliseconds of the mkdir), or the corpse of a claimer killed or out of + # disk before it could write one. Only age tells them apart; the dir's + # mtime moves on its creation and on each rename into it (a stamp or + # marker), so the age is at least that of the last write, and BSD find's + # `-mmin +1` means two minutes or more. + [ -n "$(find "$_d" -maxdepth 0 -mmin +1 2>/dev/null)" ] && _retire "$_d" + continue + fi + if [ $(( _tnow - _s )) -gt "$_bidle" ]; then + _retire "$_d" + fi + done + for _t in "$_bdir"/queue/*; do + [ -e "$_t" ] || continue + _tp=${_t##*/} + case $_tp in + *.*) kill -0 "${_tp##*.}" 2>/dev/null || rm -f "$_t" 2>/dev/null; continue ;; + esac + if ! kill -0 "$_tp" 2>/dev/null; then + rm -f "$_t" 2>/dev/null; continue + fi + _hb=""; { read -r _hb; read -r _hb; } 2>/dev/null < "$_t" || _hb="" + [ "$_hb" -ge 0 ] 2>/dev/null && [ $(( _tnow - _hb )) -gt 15 ] && rm -f "$_t" 2>/dev/null + done + # A build's first-queued record outlives its ticket by design (it is what a + # re-queue reads), so it is collected here, by the death of its cargo — never + # on the happy path, which would forfeit a re-queued build's place. Same for + # the said-once marker, a claim mutex whose holder died, and a retire tombstone. + _sweep "$_bdir/first" + _sweep "$_bdir/said" + _sweep "$_bdir/reap" + for _e in "$_bdir"/claim.*; do + [ -d "$_e" ] || continue + _n=${_e##*/}; _n=${_n#claim.} + case $_n in + *.*) kill -0 "${_n##*.}" 2>/dev/null || rm -rf "$_e" 2>/dev/null; continue ;; + esac + # Dead holder, or (a pid never written, ENOSPC) a dead cargo: either way + # nobody will release it. + _mp=""; { read -r _mp; } 2>/dev/null < "$_e/pid" || _mp="" + if { [ -n "$_mp" ] && ! kill -0 "$_mp" 2>/dev/null; } \ + || { [ -z "$_mp" ] && ! kill -0 "$_n" 2>/dev/null; }; then + mv "$_e" "$_e.$$" 2>/dev/null && rm -rf "$_e.$$" 2>/dev/null + fi + done +} + +# The slot this build already holds, if any. +_owned() { + _i=1 + while [ "$_i" -le "$_bslots" ]; do + _d="$_bdir/slot/$_i" + _p=""; { read -r _p < "$_d/pid"; } 2>/dev/null || _p="" + if [ "$_p" = "$_cargo" ]; then _bslot="$_d"; return 0; fi + _i=$((_i + 1)) + done + return 1 +} + +# Record this compile as live in $_bslot and refresh its stamp. Returns 1 if the +# slot vanished underneath us (a sibling reaped it as idle a moment ago) or +# another build re-created it in that same gap — the caller keeps waiting rather +# than compiling unslotted — and 2 if the table is unwritable, which is fail-open. +# Never re-creates the slot: a reaped slot re-made here would carry a live marker +# and no pid, which nothing can retire and nothing can claim. +_hold() { + [ -d "$_bslot/active" ] || return 1 + if ! _mark "$_bslot" || ! _stamp "$_bslot"; then + # A write that failed because the slot was retired under us — or already + # re-created by another build — is case 1, not "unwritable": fail open + # only when the slot is still this build's. Drop the marker either way. + rm -f "$_bslot/active/$$" 2>/dev/null + _p=""; { read -r _p < "$_bslot/pid"; } 2>/dev/null || _p="" + [ "$_p" = "$_cargo" ] || return 1 + return 2 + fi + _p=""; { read -r _p < "$_bslot/pid"; } 2>/dev/null || _p="" + [ "$_p" = "$_cargo" ] && return 0 + rm -f "$_bslot/active/$$" 2>/dev/null + return 1 +} + +# Write or refresh this build's ticket: line 1 is when the build FIRST queued +# (kept across a re-queue, so a build that lost its slot keeps its place), line +# 2 is the heartbeat that says a waiter is still alive behind it. Fails only +# when the ticket cannot be written. +_ticket() { + # The record names the cargo's start time as well, so a pid recycled onto a + # new build cannot inherit an old build's place — the same guard parent/ has. + [ -n "${_cstart:-}" ] || _cstart=$(ps -o lstart= -p "$_cargo" 2>/dev/null) + _f0=""; _fs="" + # `2>/dev/null` BEFORE the input redirect: redirections apply left to right, + # so the other order opens the (usually absent) file with stderr still live + # and prints `cannot open` into every cargo's output. + { read -r _f0 && IFS= read -r _fs; } 2>/dev/null < "$_bdir/first/$_cargo" || _f0="" + [ "$_fs" = "$_cstart" ] && [ "$_f0" -ge 0 ] 2>/dev/null || _f0="" + if [ -z "$_f0" ]; then + _f0=$_tnow + { printf '%s\n%s\n' "$_f0" "$_cstart" > "$_bdir/first/$_cargo.$$" \ + && mv "$_bdir/first/$_cargo.$$" "$_bdir/first/$_cargo"; } 2>/dev/null + fi + { printf '%s\n%s\n' "$_f0" "$_tnow" > "$_bdir/queue/$_cargo.$$" \ + && mv "$_bdir/queue/$_cargo.$$" "$_bdir/queue/$_cargo"; } 2>/dev/null +} + +# First come, first served: this build may take a free slot only when its ticket +# is the oldest live one (first-queued time, then pid). Tickets are keyed by +# cargo pid, so every rustc of one build shares one place in line. +_head_of_queue() { + _me=""; { read -r _me < "$_bdir/queue/$_cargo"; } 2>/dev/null || return 1 + [ "$_me" -ge 0 ] 2>/dev/null || return 1 + for _t in "$_bdir"/queue/*; do + [ -e "$_t" ] || continue + _op=${_t##*/} + case $_op in *.*) continue ;; esac # a ticket mid-rename + [ "$_op" = "$_cargo" ] && continue + _oe=""; { read -r _oe < "$_t"; } 2>/dev/null || continue + [ "$_oe" -ge 0 ] 2>/dev/null || continue + if [ "$_oe" -lt "$_me" ] || { [ "$_oe" -eq "$_me" ] && [ "$_op" -lt "$_cargo" ]; }; then + return 1 + fi + done + return 0 +} + +# Once per build, after 20s in the queue: the only sign a queued build gives is +# cargo's silent `Compiling` line, which an agent on a foreground timeout reads +# as a hang and relaunches — at the back of the line. +_say_queued() { + [ $(( _tnow - _t0 )) -ge 20 ] || return 0 + mkdir -p "$_bdir/said" 2>/dev/null && mkdir "$_bdir/said/$_cargo" 2>/dev/null || return 0 + _q=0; for _t in "$_bdir"/queue/*; do case ${_t##*/} in *.*) ;; *) [ -e "$_t" ] && _q=$((_q + 1)) ;; esac; done + # shellcheck disable=SC2016 # the backticks are prose for the reader + printf 'rustc-qos: this build is queued for a machine-wide build slot (%s builds waiting; `make build-status` shows the queue)\n' "$_q" >&2 +} + +if [ -n "$_cargo" ] && [ -z "$_exempt" ]; then + _t0="" + _noclock=0 + _cstart="" + _open="" + while :; do + # One clock reading per poll, validated once: a `date` that could not fork + # (load 400+, thousands of processes) yields "", and every age judged + # against "" — reap, claim, the wait ceiling — would be wrong in some + # direction. Skip the poll instead, and fail open after NUB_BUILD_NOCLOCK_MAX + # consecutive ones (a minute by default). + _tnow=$(_now) + if ! [ "$_tnow" -ge 1 ] 2>/dev/null; then + _noclock=$((_noclock + 1)) + [ "$_noclock" -ge "$_nomax" ] && { _bslot=""; break; } + sleep 1; continue + fi + _noclock=0 # consecutive: an intermittent miss must not add up over the queue + [ -n "$_t0" ] || _t0=$_tnow + # Each of these is a fail-open exit, taken in place rather than after + # NUB_BUILD_WAIT: the host-wide off switch, a table wiped or unwritable + # underneath us, a cargo that died while this rustc queued. + [ -e "$_bdir/off" ] && { _bslot=""; break; } + { [ -d "$_bdir/slot" ] && [ -d "$_bdir/queue" ] && [ -d "$_bdir/first" ] && [ -d "$_bdir/reap" ]; } \ + || mkdir -p "$_bdir/slot" "$_bdir/queue" "$_bdir/first" "$_bdir/reap" 2>/dev/null \ + || { _bslot=""; break; } + kill -0 "$_cargo" 2>/dev/null || { _bslot=""; break; } + _reap + if _owned; then + _hold; _rc=$? + [ "$_rc" = 0 ] && break + [ "$_rc" = 2 ] && { _bslot=""; break; } + fi + _bslot="" + _ticket || break + if _head_of_queue; then + # SIBLINGS OF ONE BUILD MUST NOT CLAIM CONCURRENTLY. Two parallel first + # compiles share one ticket, so both are head of queue at once; without + # this mutex each can claim a different slot and the build holds two — + # observed with slots=2: the yield below is racy when neither sibling's + # pid is visible yet, and a doubly-held build starves everyone else. The + # critical section is microseconds; a sibling that loses it just waits a + # loop iteration and then finds the slot _owned. A holder killed mid-claim + # leaves a mutex whose pid is dead, reclaimed by _reap. + _mx="$_bdir/claim.$_cargo" + if mkdir "$_mx" 2>/dev/null; then + { printf '%s\n' $$ > "$_mx/pid"; } 2>/dev/null + _i=1 + while [ "$_i" -le "$_bslots" ]; do + _d="$_bdir/slot/$_i" + if mkdir "$_d" 2>/dev/null; then + # Stamp and live marker BEFORE the pid: a reaper judges a slot by + # those, and a pid-bearing slot with neither would read as idle. A + # claim that cannot write is a claim on a table that cannot work. + if ! { _stamp "$_d" && mkdir -p "$_d/active" 2>/dev/null && _mark "$_d" \ + && { printf '%s\n' "$_cargo" > "$_d/pid"; } 2>/dev/null; }; then + _retire "$_d"; _bslot=""; _open=1; break + fi + # Backstop for the same hazard across a mutex reclaim: keep only + # the lowest slot this build owns, yield any extra. + _bslot="" + if _owned && [ "$_bslot" = "$_d" ]; then break; fi + _retire "$_d"; _bslot="" + break + fi + _i=$((_i + 1)) + done + rm -rf "$_mx" 2>/dev/null + fi + [ -n "$_bslot" ] && break + [ -n "$_open" ] && { _bslot=""; break; } + fi + if [ $(( _tnow - _t0 )) -ge "$_bwait" ]; then + _bslot=""; break # pathological wait: fail open, never stall a build + fi + _say_queued + sleep 1 + done + rm -f "$_bdir/queue/$_cargo" 2>/dev/null +fi +# --------------------------------------------------------------- rustc tokens + # Retry ceiling x the sleep below bounds a wait at ~10 min. A rustc that waits # that long has hit something pathological (a leaked token dir whose holder pid # got recycled, say), so degrade to unthrottled rather than stall a build. -if [ "${_limit:-0}" -gt 0 ] 2>/dev/null && mkdir -p "$_sem" 2>/dev/null; then +if [ "$_limit" -gt 0 ] && mkdir -p "$_sem" 2>/dev/null; then _tries=0 - while [ "$_tries" -lt "${NUB_RUSTC_SEM_TRIES:-1500}" ]; do + while [ "$_tries" -lt "$_tries_max" ]; do _i=1 while [ "$_i" -le "$_limit" ]; do _d="$_sem/$_i" # mkdir is the atomic test-and-set; the pid inside is only for reclaim. if mkdir "$_d" 2>/dev/null; then - printf '%s\n' $$ > "$_d/pid" 2>/dev/null || true + { printf '%s\n' $$ > "$_d/pid"; } 2>/dev/null || true _slot="$_d" break fi @@ -92,12 +519,13 @@ if [ "${_limit:-0}" -gt 0 ] 2>/dev/null && mkdir -p "$_sem" 2>/dev/null; then # still prints the REDIRECTION failure on the shell's own stderr, and this # races constantly (a holder that has mkdir'd but not yet written its pid). # That noise would interleave into every cargo's stderr machine-wide. + # Rename before rm, as _retire does: two waiters can judge one holder dead. _p="" if [ -r "$_d/pid" ]; then { read -r _p < "$_d/pid"; } 2>/dev/null || _p="" fi if [ -n "$_p" ] && ! kill -0 "$_p" 2>/dev/null; then - rm -rf "$_d" 2>/dev/null || true + mv "$_d" "$_sem/.reap.$_i.$$" 2>/dev/null && rm -rf "$_sem/.reap.$_i.$$" 2>/dev/null fi _i=$((_i + 1)) done @@ -106,22 +534,54 @@ if [ "${_limit:-0}" -gt 0 ] 2>/dev/null && mkdir -p "$_sem" 2>/dev/null; then sleep 0.4 done fi +# The inner wrapper (see RE-ENTRANCY) inherits this shell's decision — a slot +# and token held, a layer disabled, or a wait already failed open — rather +# than running the protocol again. +NUB_RUSTC_SEM_HELD=1 +export NUB_RUSTC_SEM_HELD -# Uncontended, or fail-open: nothing to release, so exec and drop this shell. -if [ -z "$_slot" ]; then +# Nothing held (exempt, or failed open): nothing to release, so exec and drop +# this shell. +if [ -z "$_slot" ] && [ -z "$_bslot" ]; then # shellcheck disable=SC2086 # $_qos word-splits deliberately (empty, or the clamp) exec $_qos "$@" fi -# Holding a token means this shell must OUTLIVE rustc to release it, so no exec. -NUB_RUSTC_SEM_HELD=1 -export NUB_RUSTC_SEM_HELD -trap 'rm -rf "$_slot" 2>/dev/null' EXIT +# Holding a token or a slot means this shell must OUTLIVE rustc, so no exec: the +# token is released here, and the slot's stamp is refreshed at compile END as +# well as start. The end stamp is load-bearing: a compile longer than +# NUB_BUILD_IDLE would otherwise leave its build looking idle the instant it +# finished, and a waiter polling in the gap before cargo's next rustc would take +# the slot from a build in full flight (the harness's idle scenario runs a +# compile longer than its window for exactly this; `aube` alone outlasts the +# 120s default under load). +# shellcheck disable=SC2329 # invoked from the traps below +_release() { + [ -n "$_slot" ] && rm -rf "$_slot" 2>/dev/null + if [ -n "$_bslot" ]; then + # Stamp FIRST, marker second. A reaper scans the markers and reads the + # stamp only once it finds none live, so a reaper that sees this marker + # gone is guaranteed to see the fresh stamp. The other order let a waiter + # polling in the same second retire a slot whose compile had just ended + # (caught by the idle scenario under load). + # Only OUR slot: the path may by now name a slot retired and re-claimed by + # another build (a SIGKILLed cargo leaves its wrappers running), whose + # stamp is not ours to refresh. + _p=""; { read -r _p < "$_bslot/pid"; } 2>/dev/null || _p="" + _tnow=$(_now) + [ "$_p" = "$_cargo" ] && [ "$_tnow" -ge 1 ] 2>/dev/null && _stamp "$_bslot" + rm -f "$_bslot/active/$$" 2>/dev/null + fi + return 0 +} +trap '_release' EXIT # shellcheck disable=SC2086 $_qos "$@" & _child=$! # Forward termination, so killing the wrapper kills the rustc it owns. Without # this the token would be released while the compile it guards still runs. -trap 'kill -TERM "$_child" 2>/dev/null; rm -rf "$_slot" 2>/dev/null; exit 143' INT TERM +trap 'kill -TERM "$_child" 2>/dev/null; _release; exit 143' INT TERM wait "$_child" exit $? + +} diff --git a/tests/build-slots/run.sh b/tests/build-slots/run.sh new file mode 100755 index 000000000..f6be29f66 --- /dev/null +++ b/tests/build-slots/run.sh @@ -0,0 +1,311 @@ +#!/bin/sh +# shellcheck disable=SC2016,SC2329 # assertions are eval-ed expressions by design +# Exercises the build-slot layer of scripts/rustc-qos.sh without cargo, rustc +# or a loaded host: a fake `cargo` (a tiny C program that forks a shell script +# and waits — it must stay alive as an ANCESTOR, and it must be NAMED cargo, +# which a shell script cannot be on darwin where `ps comm` reports the +# interpreter) drives a fake `rustc` that only logs start/end and sleeps. +# Every scenario asserts on the ordering of those events, and the first one +# is the positive control: a second build must NOT start while the first +# compiles, so a refactor that fails the layer open goes red here. +# +# tests/build-slots/run.sh # all scenarios, ~3 min on a CI runner +# tests/build-slots/run.sh fifo kill # a subset +# +# POSIX sh; needs a C compiler on PATH (cc). State lives under a private +# NUB_BUILD_SEM_DIR so the machine's real governor is untouched. +set -u +here=$(cd "$(dirname "$0")" && pwd) +W="$here/../../scripts/rustc-qos.sh" +T=${TMPDIR:-/tmp}/build-slots-test.$$ +mkdir -p "$T/bin" "$T/log" +trap 'pkill -f "$T" 2>/dev/null; rm -rf "$T"' EXIT INT TERM + +cat > "$T/cargo.c" <<'C' +#include +#include +#include +int main(int argc, char **argv) { + pid_t p = fork(); + if (p == 0) { argv[0] = "/bin/sh"; execv("/bin/sh", argv); _exit(127); } + int st = 0; waitpid(p, &st, 0); + return WIFEXITED(st) ? WEXITSTATUS(st) : 128 + WTERMSIG(st); +} +C +# A missing compiler is a skip on a dev box and a FAILURE in CI, where a skip +# would read as a green governor job with zero coverage. +cc -o "$T/bin/cargo" "$T/cargo.c" || { echo "SKIP: no C compiler"; [ -n "${CI:-}" ] && exit 1; exit 0; } +cp "$T/bin/cargo" "$T/bin/rust-analyzer" + +# fake rustc: $1 build name, $2 seconds, $3 exit status (default 0). It shares +# the wrapper's stderr, so its own `date` is silenced: SIGKILLed mid-call under +# a parent that ignores SIGPIPE (the CI runner) it would print `Broken pipe` +# into the file the stderr assertion reads. +cat > "$T/bin/rustc" </dev/null)" "\$1" >> "$T/log/events" +sleep "\$2" +printf '%s %s end\n' "\$(/bin/date +%s 2>/dev/null)" "\$1" >> "$T/log/events" +exit "\${3:-0}" +RUSTC +# The fakes name /bin/date so the noclock scenario can starve the WRAPPER of a +# clock through PATH without blinding the event log. +chmod +x "$T/bin/rustc" + +# fake build: name, compiles, parallelism, seconds per compile, [seconds +# before the first compile — the cargo exists, and so holds its pid, before +# it queues] +cat > "$T/build.sh" <> "$T/log/events" +[ -z "\${5:-}" ] || sleep "\$5" +i=0 +while [ \$i -lt \$n ]; do + j=0 + while [ \$j -lt \$par ] && [ \$i -lt \$n ]; do + "$W" "$T/bin/rustc" "\$name" "\$dur" 2>>"$T/log/wrapper.err" & + j=\$((j+1)); i=\$((i+1)) + done + wait +done +printf '%s %s cargo-end\n' "\$(/bin/date +%s)" "\$name" >> "$T/log/events" +BUILD + +# The scenarios exercise the MECHANISM, so they pin one slot; the machine +# default is 2 (see rustc-qos.sh) and the exempt/disabled paths are tested +# explicitly below. +export NUB_BUILD_SEM_DIR="$T/sem" NUB_RUSTC_SEM_DIR="$T/rsem" NUB_BUILD_WAIT=40 NUB_BUILD_SLOTS=1 +unset NUB_BUILD_FG NUB_BUILD_IDLE NUB_BUILD_MAXCOMPILE NUB_BUILD_NOCLOCK_MAX +fails=0 +# A scenario's leftovers (a build tree that outlived a kill) must not log into +# the next one's events, so reset takes down every fake process first. +reset() { + pkill -9 -f "$T/build.sh" 2>/dev/null; pkill -9 -f "$T/bin/rustc" 2>/dev/null; sleep 1 + : > "$T/log/events"; rm -rf "$T/sem" "$T/rsem" +} +: > "$T/log/wrapper.err" +build() { "$T/bin/cargo" "$T/build.sh" "$@"; } +# seconds from the log's first event to the first / last " " event +at() { awk -v n="$1" -v k="$2" 'NR==1{t0=$1} $2==n && $3==k {print $1-t0; exit}' "$T/log/events"; } +last() { awk -v n="$1" -v k="$2" 'NR==1{t0=$1} $2==n && $3==k {t=$1-t0} END{print t}' "$T/log/events"; } +check() { + if eval "$2" 2>/dev/null; then echo " ok $1"; else echo " FAIL $1"; fails=$((fails + 1)); fi +} +timeline() { awk 'NR==1{t0=$1} {printf "t+%ds %s %s; ", $1-t0, $2, $3}' "$T/log/events"; echo; } +want=" $* " +run() { [ "$want" = " " ] || case $want in *" $1 "*) return 0 ;; *) return 1 ;; esac; } + +if run serialize; then + echo "serialize: B must not start until A's cargo has exited (positive control)" + reset; build A 4 2 3 & sleep 1; build B 2 2 2 & wait; timeline + # Relational, not a wall-clock literal: with 4 compiles of 3s in pairs the + # last start is ~3s after the first; over-throttled to one at a time it is + # ~9s. The bound sits at the midpoint so a slow runner has 3s of slack. + check "A's compiles ran in pairs (not over-throttled to one at a time)" '[ $(( $(last A start) - $(at A start) )) -le 6 ]' + check "B did start" '[ -n "$(at B start)" ]' + check "B started only after A ended" '[ "$(at B start)" -ge "$(at A cargo-end)" ]' +fi + +if run fifo; then + echo "fifo: three builds leave in the order they arrived" + # B compiles 2s so that with no wrapper at all C (arriving 1s after B) + # would start during B's compile: arrival order alone must not satisfy this. + reset; build A 2 2 3 & sleep 1; build B 1 1 2 & sleep 1; build C 1 1 1 & wait; timeline + check "B before C" '[ "$(at B start)" -lt "$(at C start)" ]' + check "C after B ended" '[ "$(at C start)" -ge "$(at B end)" ]' +fi + +if run kill; then + echo "kill: a SIGKILLed holder releases the slot within seconds" + reset; build K 6 2 3 & sleep 1; build B 1 1 1 & sleep 3 + # the whole build tree: the cargo shim (slot holder), its build script, its wrappers + pkill -9 -f "$T/build.sh K " 2>/dev/null; pkill -9 -f "$T/bin/rustc K " 2>/dev/null + wait 2>/dev/null; timeline + # The kill lands at t+4; without dead-holder reclaim B waits for K's 6 compiles. + check "B started within a few seconds of the kill" '[ "$(at B start)" -le 12 ]' +fi + +if run idle; then + echo "idle: a holder with no live compile for NUB_BUILD_IDLE yields; a holder mid-build is never robbed; a re-queue keeps its place" + reset + cat > "$T/idle.sh" <> "$T/log/events" +"$W" "$T/bin/rustc" A 1 2>>"$T/log/wrapper.err"; sleep 8 +"$W" "$T/bin/rustc" A 5 2>>"$T/log/wrapper.err" +printf '%s A cargo-end\n' "\$(date +%s)" >> "$T/log/events" +IDLE + # C's cargo is launched FIRST (lowest pid) but compiles only from t+2; A + # compiles 1s at t+0 and idles 8s; B queues at t+1, D at t+5. B reclaims the + # slot at ~t+6 and holds it to ~t+12 (two 5s compiles in a pair), so A + # re-queues (t+9) while B still holds: at B's release the line is A (first + # queued t+0), C (t+2), D (t+5), and A must go first — unless a re-queue + # loses its place, in which case the tie falls to C's lower pid. C's compiles + # are 6s under a 4s window: a holder whose compile OUTLASTS the window is + # exactly the build a stale start-stamp would expose. Mutation-checked: + # dropping the end-stamp lets A steal from C (the +15 bound); ignoring live + # markers, or skipping _hold's pid check, lets D steal (the D-after-C + # check). A 3s compile caught none of these. + NUB_BUILD_IDLE=4 build C 3 1 6 2 & + NUB_BUILD_IDLE=4 "$T/bin/cargo" "$T/idle.sh" & sleep 1 + NUB_BUILD_IDLE=4 build B 2 2 5 & sleep 4 + NUB_BUILD_IDLE=4 build D 1 1 1 & wait; timeline + # Relational: B may start once A's first compile is 4s idle, plus polling + # (observed +5..6). Without idle reclaim B waits for A's cargo to exit, ~30s. + check "B took the slot while A idled" '[ "$(at B start)" -le $(( $(at A end) + 4 + 5 )) ]' + check "A's re-queue kept its place: its second compile ran before C's first" '[ "$(last A start)" -lt "$(at C start)" ]' + # Correct: C's third start is 12s after its first. A steal costs C at least + # D's compile plus polling; 15 leaves 3s of slack. + check "C's three compiles were never interrupted (a compile longer than the window is not idle)" '[ "$(last C start)" -le $(( $(at C start) + 15 )) ]' + check "D never ran before C's cargo exited" '[ "$(at D start)" -ge "$(at C cargo-end)" ]' +fi + +if run nested; then + echo "nested: an inner cargo inherits the outer build's slot instead of queueing behind it" + reset + cat > "$T/nested.sh" <> "$T/log/events" +"$W" "$T/bin/rustc" A 1 2>>"$T/log/wrapper.err" +"$T/bin/cargo" "$T/build.sh" Ai 2 2 2 +printf '%s A cargo-end\n' "\$(date +%s)" >> "$T/log/events" +NESTED + "$T/bin/cargo" "$T/nested.sh" & sleep 1; build B 1 1 1 & wait; timeline + # Bounded by A's 1s compile plus polling, not by cargo-end (which the inner's + # synchronous return always precedes): a queued inner would sit until fail-open. + check "inner ran while outer held" '[ "$(at Ai start)" -le $(( $(at A start) + 4 )) ]' + check "B waited for the outer" '[ "$(at B start)" -ge "$(at A cargo-end)" ]' +fi + +if run orphan; then + echo "orphan: a queued wrapper killed while its cargo lives on must not wedge the head of the queue" + reset + cat > "$T/orphan.sh" <> "$T/log/events" +"$W" "$T/bin/rustc" O 1 2>>"$T/log/wrapper.err" & w=\$! +sleep 2; kill -9 "\$w" 2>/dev/null +sleep 25 +printf '%s O cargo-end\n' "\$(date +%s)" >> "$T/log/events" +ORPHAN + build A 1 1 4 & sleep 1; "$T/bin/cargo" "$T/orphan.sh" & sleep 1; build B 1 1 1 & wait; timeline + check "B ran before O's cargo exited (stale ticket ignored)" '[ "$(at B start)" -lt "$(at O cargo-end)" ]' +fi + +if run deadcargo; then + echo "deadcargo: a wrapper whose cargo was killed while it queued fails open at once, not after NUB_BUILD_WAIT" + reset; build A 1 1 8 & sleep 1; build Q 1 1 1 & sleep 2 + # Only Q's cargo shim: its build script and queued wrapper live on, orphaned. + pkill -9 -f "^$T/bin/cargo $T/build.sh Q " 2>/dev/null + wait 2>/dev/null; timeline + check "Q's orphaned rustc ran before A ended (fail-open in place)" '[ "$(at Q start)" -lt "$(at A end)" ]' +fi + +if run torn; then + echo "torn: an empty stamp and an empty live marker (what a torn read of an in-place write sees) must not cost a holder its slot" + reset; build A 1 1 8 & sleep 1; build B 1 1 1 & sleep 1 + # The race is hard to schedule but its state is trivial to plant: truncate + # A's stamp and its live marker mid-compile, and age the slot dir past the + # stampless rule. A reader that judges an unreadable marker dead, or reads + # the stamp before the markers, retires the slot and B steals it. + n=0; for m in "$T"/sem/slot/1/active/*; do [ -e "$m" ] || continue; : > "$m"; n=$((n + 1)); done + : > "$T/sem/slot/1/stamp" + touch -t 202001010000 "$T/sem/slot/1" + wait; timeline + # Positive control: the plant is a no-op if A never held slot 1. + check "A held slot 1 with a live marker to truncate" '[ "$n" -gt 0 ]' + check "B did not take A's slot while A compiled" '[ "$(at B start)" -ge "$(at A end)" ]' +fi + +if run noclock; then + echo "noclock: a wrapper that cannot read the clock fails open after NUB_BUILD_NOCLOCK_MAX CONSECUTIVE misses, not after NUB_BUILD_WAIT" + reset; mkdir -p "$T/noclock" "$T/flipclock" + printf '#!/bin/sh\nexit 1\n' > "$T/noclock/date"; chmod +x "$T/noclock/date" + # Fails every other call, so the longest run of consecutive misses is one. + cat > "$T/flipclock/date" </dev/null || echo 0); echo \$((c + 1)) > "$T/flipclock/n" +[ \$((c % 2)) -eq 0 ] && exit 1 +exec /bin/date "\$@" +FLIP + chmod +x "$T/flipclock/date" + # Threshold: the fifth miss breaks before sleeping, so four sleeps by + # construction; the wait ceiling is 40s. + PATH="$T/noclock:$PATH" NUB_BUILD_NOCLOCK_MAX=5 build N 1 1 1 & wait; timeline + check "N compiled after the valve (four sleeps), not at the wait ceiling" '[ "$(at N start)" -ge 4 ] && [ "$(at N start)" -le 20 ]' + # Consecutive vs cumulative: A holds the slot for 12s while B's clock misses + # every other poll. Counted cumulatively, B's third miss (about poll 5) would + # fail it open onto A's compile; counted consecutively the run never passes + # one, and B waits for A. + reset; rm -f "$T/flipclock/n" + build A 1 1 12 & sleep 1; PATH="$T/flipclock:$PATH" NUB_BUILD_NOCLOCK_MAX=3 build B 1 1 1 & wait; timeline + check "B waited for A although its clock missed every other poll" '[ "$(at B start)" -ge "$(at A end)" ]' + check "B's clock was the alternating fake (it was consulted at least six times)" '[ "$(cat "$T/flipclock/n")" -ge 6 ]' +fi + +if run twoslots; then + echo "twoslots: with two slots, a second build overlaps the first instead of queueing" + reset + # A's two parallel first compiles share one ticket and race the claim; the + # per-build mutex must keep A on ONE slot so B can take the other. Without + # it this scenario hangs B until A exits (observed before the mutex). + NUB_BUILD_SLOTS=2 build A 2 2 4 & sleep 1; NUB_BUILD_SLOTS=2 build B 2 2 2 & wait; timeline + check "B ran alongside A" '[ "$(at B start)" -lt "$(at A cargo-end)" ]' + check "A's pair still ran in parallel" '[ "$(last A start)" -lt "$(at A end)" ]' +fi + +if run tokens; then + echo "tokens: NUB_RUSTC_LIMIT=1 serializes a slot holder's parallel compiles" + reset; NUB_RUSTC_LIMIT=1 build A 4 2 2 & wait; timeline + # In pairs the last start is ~2s after the first; one token at a time, ~6s. + check "A's compiles ran one at a time" '[ $(( $(last A start) - $(at A start) )) -ge 5 ]' +fi + +if run exit; then + echo "exit: rustc's exit status reaches cargo on the held path and the exec path" + reset + "$T/bin/cargo" -c "\"$W\" \"$T/bin/rustc\" X 1 3 2>>\"$T/log/wrapper.err\"; echo \$? > \"$T/log/rc.held\"" + NUB_BUILD_SLOTS=0 NUB_RUSTC_LIMIT=0 "$T/bin/cargo" -c "\"$W\" \"$T/bin/rustc\" Y 1 3 2>>\"$T/log/wrapper.err\"; echo \$? > \"$T/log/rc.exec\"" + check "held path returned 3" '[ "$(cat "$T/log/rc.held")" = 3 ]' + check "exec path returned 3" '[ "$(cat "$T/log/rc.exec")" = 3 ]' +fi + +if run off; then + echo "off: the host-wide off switch releases a build already queued" + reset; build A 1 1 6 & sleep 1; build B 1 1 1 & sleep 2 + touch "$T/sem/off"; wait; timeline + check "B started once the switch was set, before A ended" '[ "$(at B start)" -lt "$(at A end)" ]' +fi + +if run probe; then + echo "probe: rustc -vV never queues" + reset; build A 1 1 4 & sleep 1 + s=$(date +%s); "$T/bin/cargo" -c "\"$W\" /bin/echo -vV >/dev/null 2>>\"$T/log/wrapper.err\""; e=$(( $(date +%s) - s )); wait + check "probe returned immediately ($e s)" '[ "$e" -le 1 ]' +fi + +if run exempt; then + echo "exempt: NUB_BUILD_FG=1, rust-analyzer ancestry and NUB_BUILD_SLOTS=0 all run alongside a holder" + reset; build A 1 1 5 & sleep 1 + NUB_BUILD_FG=1 build F 1 1 1 & + "$T/bin/rust-analyzer" -c "\"$T/bin/cargo\" \"$T/build.sh\" R 1 1 1" & + NUB_BUILD_SLOTS=0 build Z 1 1 1 & wait; timeline + check "FG build overlapped A" '[ "$(at F start)" -lt "$(at A end)" ]' + check "rust-analyzer build overlapped A" '[ "$(at R start)" -lt "$(at A end)" ]' + check "slots=0 build overlapped A" '[ "$(at Z start)" -lt "$(at A end)" ]' +fi + +# Independent of every scenario above: the wrapper sits in front of every rustc +# on the machine, so anything it writes to stderr lands in every cargo's output. +# Both stderr-noise bugs found on this script (`Illegal number`, `cannot open`) +# were invisible to the timeline assertions; this is the assertion that sees them. +# The one deliberate line — the once-per-build "queued" notice — is exempt. +echo "stderr: the wrapper wrote nothing but its queued notice to stderr across the whole run" +noise=$(grep -v '^rustc-qos: ' "$T/log/wrapper.err") +if [ -n "$noise" ]; then + echo " FAIL wrapper stderr:"; printf '%s\n' "$noise" | sed 's/^/ /' | head -10 + fails=$((fails + 1)) +else + echo " ok wrapper stderr clean" +fi + +echo +if [ "$fails" = 0 ]; then echo "build-slots: all scenarios passed"; else echo "build-slots: $fails assertion(s) FAILED"; fi +exit "$fails"