Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1655,6 +1655,33 @@ the current one and `final` supersedes them all. The partials exist because
killed or times out is exactly the run whose startup timings you want, and it
was precisely the one that left no trace of them at all.

### Skipped ticks — the usage-gate pause row (#160)

A tick the weekly-budget pace gate pauses (usage-gate exit 10) still writes one
`stage: final` row, from the runner's exit-10 path over an empty trace:

```json
"skipped": "usage-gate", "skipReason": "<the gate's own PAUSE line, verbatim>", "outcome": "skipped", "exitCode": 10
```

`exitCode` is the GATE's 10, the same way the preflight-abort row records
preflight's 12 — the runner itself still exits 0 because a pause is not a
failure. Both skip fields are **absent** — not null — on every other row, so a
consumer keys on the field existing at all and pre-#160 records read unchanged.
Before this row existed a paused stretch left nothing in the file, and the
dashboard drew nine consecutive gated ticks as a dead cron. A config REFUSAL
(usage-gate exit 2) is NOT a skip: the tick aborts loudly and writes no row —
broken config must never render as pacing.

### How the file reaches main

The runners append rows and never push. The hourly `refresh-human-queue` cron
stages `metrics/runs.jsonl` beside the snapshot it already commits straight to
main, publishing when EITHER file moved. That cron carries it because it is
data-only and never usage-gated — the one committer still awake during a pause,
which is exactly when skip rows are written and nothing else runs. A skip row is
therefore visible to the dashboard within about an hour of its gated tick.

### Live token spend — and the one number that is not knowable

`run-metrics` reads tokens from the terminal `result` event, so a killed run
Expand Down
35 changes: 27 additions & 8 deletions campaign-run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -87,14 +87,32 @@ fi
# --- weekly-budget pace gate: skip this tick when usage is over the ceiling or inside the BAU
# headroom band under the linear burn toward the reset — the crons hold ~USAGE_HEADROOM_PCT points
# BEHIND pace so interactive work keeps standing budget (#158). `usage-gate` reads
# /api/oauth/usage itself; exit 10 means PAUSE (log it, exit 0). It is INERT when it cannot read
# usage and no fallback is set — it prints OK and we run. Any OTHER non-zero exit is a config
# REFUSAL (the retired USAGE_SLACK_PCT still set: exit 2, reason on stderr, captured into the
# log): the tick must not run on config the gate refused to read, so propagate the failure — a
# refusal is neither a run nor a pause. ---
# /api/oauth/usage itself; exit 10 means PAUSE (record one skip row, exit 0). It is INERT when it
# cannot read usage and no fallback is set — it prints OK and we run. Any OTHER non-zero exit is a
# config REFUSAL (the retired USAGE_SLACK_PCT still set: exit 2, reason on stderr, captured into
# the log): the tick must not run on config the gate refused to read, so propagate the failure — a
# refusal is neither a run nor a pause, and it writes NO row. ---
_ug="$(pr-review-report usage-gate 2>&1)"; _ugrc=$?
echo "$(date -u +%FT%TZ) usage-gate: $_ug" >> "$LOG"
[ "$_ugrc" -eq 10 ] && exit 0
if [ "$_ugrc" -eq 10 ]; then
# A paused tick still writes its metrics/runs.jsonl row (#160): the dashboard reads runs from
# that file, and a pause that wrote nothing rendered as a dead stretch indistinguishable from a
# broken cron. Same shape as the preflight abort below — an empty trace, so the record's shape
# still comes from `run-metrics` and no second place knows what a runs.jsonl line looks like.
# The row records the GATE's exit 10 (as the preflight row records preflight's 12) plus the
# gate's own line, verbatim; this script still exits 0 because a pause is not a failure. The row
# reaches origin/main via the hourly refresh-human-queue cron, which stages this file — the one
# committer still awake during a pause.
TS="$(date -u +%Y%m%dT%H%M%SZ)"
RUNLOG="$RUNDIR/$TS.jsonl"
mkdir -p "$RUNDIR" "$DIR/metrics"
: > "$RUNLOG"
pr-review-report run-metrics "$RUNLOG" \
--run-id "$TS" --role producer --model "$MODEL" --exit-code 10 \
--skipped usage-gate --skip-reason "$_ug" \
>> "$DIR/metrics/runs.jsonl" 2>/dev/null || true
exit 0
fi
if [ "$_ugrc" -ne 0 ]; then
echo "$(date -u +%FT%TZ) campaign run ABORTED: usage-gate refused its config (exit $_ugrc) — fix cron.env" >> "$LOG"
exit "$_ugrc"
Expand Down Expand Up @@ -313,8 +331,9 @@ fi
echo "$(date -u +%FT%TZ) campaign run END (exit=$rc, trace=$RUNLOG, err=$ERRLOG)" >> "$LOG"

# Persist per-run metrics BEFORE the next run's rotation deletes this trace.
# Appends one enriched JSON line to metrics/runs.jsonl (committed periodically,
# never from here — the cron does not push). Best-effort: never fail the run on it.
# Appends one enriched JSON line to metrics/runs.jsonl (committed+pushed to main by the hourly
# refresh-human-queue cron, never from here — this cron does not push). Best-effort: never fail
# the run on it.
# `run-metrics` emits the whole enriched record itself now, including `outcome` — which it derives
# with the same typed classifier the fallback loop uses, so the metrics line and the fallback
# decision can never disagree about whether a run was quota-limited.
Expand Down
Loading
Loading