Record usage-gate pauses as typed skip rows in metrics/runs.jsonl, published hourly

[thedavidmeister]

Closes #160

A usage-gate pause (exit 10) left NO row in metrics/runs.jsonl — the runners logged one line and exited 0 — so the rain-org-health metrics page, which fetches that file raw from main, drew the current nine-plus-tick pause as a dead stretch indistinguishable from a broken cron. Human report, verbatim: "roh metrics page is not showing runs that rate limited on usage, so we just have long dead stretches in the chart."

The row (the contract a renderer builds against)

The pause path mirrors the preflight-abort just below it — an empty trace through pr-review-report run-metrics, so no second place knows what a runs.jsonl line looks like. One real emitted row (dev build at this head):

{"trace":"/tmp/empty-trace-160.jsonl","stage":"final","toolCalls":0,"startupToolCalls":0,"startupPct":0.0,"wakeupCalls":0,"firstMutationIndex":null,"bootMs":null,"ttlMs":null,"startupMs":null,"durationMs":0,"numTurns":0,"tokensIn":0,"tokensOut":0,"cacheRead":0,"cacheCreation":0,"costUsd":0.0,"rateLimits":{},"unreadableFiles":[],"commandsNotFound":[],"missingTools":[],"infraDown":false,"infraReason":"","infraRootCause":"","runId":"20260731T130001Z","role":"producer","model":"claude-fable-5","exitCode":10,"outcome":"skipped","skipped":"usage-gate","skipReason":"PAUSE: 91% of the weekly budget used (endpoint) — at/over the 90% ceiling"}

(In production trace is $RUNDIR/<runId>.jsonl — the empty trace the runner creates for the row, same relationship every other row has with its trace.)

• "skipped": "usage-gate" + "skipReason": "<the gate's own line, verbatim>" — the typed discriminant a consumer keys on. Both fields are ABSENT (not null) on every other row, so pre-usage-gate pauses leave no runs.jsonl row — paused stretches read as a dead cron everywhere downstream #160 records read unchanged.
• exitCode is the GATE's 10, the same way the preflight-abort row records preflight's 12; the runner itself still exits 0 (a pause is not a failure).
• outcome is a typed "skipped" — without it, exit 10 over an empty trace classifies error, which is exactly the dead-cron rendering this exists to correct.
• The two flags arrive together or not at all (clap requires both ways), so no runner edit can half-supply the pair.
• A config REFUSAL (usage-gate exit 2) is NOT a skip: the tick still aborts loudly with the gate's exit and writes NO row. Behavioral tests pin that boundary from the scripts' outside.

Visibility: how a skip row reaches main within the hour

metrics/runs.jsonl had NO committer: the runners append and never push ("committed periodically" meant hand-run chore(metrics): commits, last on 2026-07-28). During a pause that is fatal — the paused runs are the only writers, and nothing commits.

Chosen mechanism: the hourly refresh-human-queue cron now stages metrics/runs.jsonl beside the snapshot it already commits straight to main, and publishes when EITHER file moved. Justification:

• It is the repo's one existing straight-to-main data committer, with the sync/race/replay choreography already built and behaviourally tested — extending its git add by one tracked file is the smallest possible delta.
• It is data-only and never usage-gated, so it is the one committer still awake during a pause — which is precisely when skip rows are written and nothing else runs. Hourly cadence gives ~1h worst-case visibility.
• The alternative (pushing from the runners' pause path) would give the model runners a push path they deliberately do not have ("this cron does not push"), and would add a new git choreography in a second place instead of reusing the tested one.
• The {ts, counts} history rollup stays gated on the SNAPSHOT having changed — a metrics-only publish appends no history line, or an idle queue would grow one identical rollup per skip row.

Side effect at deploy, deliberate: the first hourly tick also publishes the ordinary run rows accrued locally since the last hand commit. That is the file's normal content finally getting its committer — NOT a back-fill of skip rows. The ~9 log-only paused ticks stay log-only; skip rows start at deploy.

QA

• Discriminating tests: skip_row_tests::{a_skip_row_carries_the_gate_and_its_reason_verbatim, an_unskipped_row_omits_both_skip_fields_entirely, a_skip_classifies_skipped_where_the_bare_exit_code_would_read_error, a_refusal_exit_is_never_skipped_without_the_flag} (contract), run_metrics_trace (CLI: skip form parses; either flag alone is refused), tests/usage_gate_skip.rs::{a_paused_producer_tick_writes_one_skip_row_and_exits_zero, a_paused_vetter_tick_writes_one_skip_row_and_exits_zero, a_refused_producer_tick_aborts_loudly_and_writes_no_row, a_refused_vetter_tick_aborts_loudly_and_writes_no_row} (real scripts as processes, only usage-gate stubbed — run-metrics is the REAL binary), refresh_human_queue::{a_metrics_only_append_publishes_without_a_history_line, a_staged_but_uncommitted_append_is_still_published} (real git remote; the second is the CodeRabbit round — probes now diff against HEAD so a change left staged by a failed commit still publishes, and the test failed on the pre-fix code). Verified on base (main @ 9785128, new test files copied onto an untouched base clone): a_paused_producer_tick… and a_paused_vetter_tick… FAIL (base writes no row) and a_metrics_only_append… FAILS (base commits nothing, origin unmoved); the unit/CLI tests cannot compile on base (the flags don't exist — the contract is new surface). The two refusal tests pass on base by design: they pin the boundary that must NOT move, and are the named killers of the swap mutant below.
• Mutations applied: (1) final_record skip-insert dropped (let _ = skip;) → a_skip_row_carries_the_gate_and_its_reason_verbatim + a_paused_producer_tick_writes_one_skip_row_and_exits_zero FAILED; (2) classify_outcome skip-precedence dropped (let _ = skipped;) → a_skip_classifies_skipped_where_the_bare_exit_code_would_read_error FAILED; (3) skip/refusal SWAPPED in campaign-run.sh (row on exit 2, bare abort on 10) → a_refused_producer_tick_aborts_loudly_and_writes_no_row + a_paused_producer_tick_writes_one_skip_row_and_exits_zero FAILED; (4) fields inserted as null when unskipped → an_unskipped_row_omits_both_skip_fields_entirely FAILED. Each reverted; full suite green at head.
• Oracle: field names and semantics verbatim from usage-gate pauses leave no runs.jsonl row — paused stretches read as a dead cron everywhere downstream #160's stated contract; skipReason fixture is the gate's real ceiling-pause line from usage_gate_decide (em-dash and all), asserted byte-identical through the runner's $_ug capture; expected JSON asserted as hand-written literals, never re-derived through final_record; the preflight-abort path is the modeled row shape, per the runners' own stated principle.
• Category check: usage-gate pauses leave no runs.jsonl row — paused stretches read as a dead cron everywhere downstream #160 asks (1) typed skip fields on run-metrics, (2) both runners' exit-10 paths emit, (3) a skip row visible on main within ~1h during a long pause, (4) refusal ≠ skip, (5) no back-fill. Covered: 1 (contract + unit/CLI tests), 2 (both runners, role producer/vetter, behavioral tests), 3 (hourly refresh-human-queue stages the file, publish-on-either-change, behavioral test on a real remote), 4 (exit 2 aborts loudly, no row, tested on both runners), 5 (this diff touches no data file — rows start at deploy).

🤖 Generated with Claude Code

[coderabbitai]

Walkthrough

Usage-gate pauses now produce skipped metrics rows with exit code 10 and empty traces. Configuration refusals remain unrecorded with exit code 2. The hourly refresh cron publishes metrics-only changes.

Changes

Usage-gate metrics flow

Layer / File(s)	Summary
Skipped metrics contract pr-review-report-rs/src/main.rs	The metrics CLI accepts paired skip options. Explicit skips serialize as skipped outcomes with verbatim skipReason values. Existing outcomes and refusal handling remain unchanged.
Runner pause recording campaign-run.sh, review-run.sh, pr-review-report-rs/tests/usage_gate_skip.rs	Pause paths create empty traces, append producer or vetter skip rows, and exit successfully. Refusal paths return exit code 2 without appending rows.
Metrics-only refresh publication refresh-human-queue.sh, pr-review-report-rs/tests/refresh_human_queue.rs, README.md	The refresh cron commits and pushes metrics changes when the queue snapshot is unchanged. Tests and documentation cover the pause-row format and publication behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Runner
  participant usage-gate
  participant pr-review-report
  participant metrics/runs.jsonl
  participant refresh-human-queue
  Runner->>usage-gate: Evaluate usage gate
  usage-gate-->>Runner: PAUSE with exit code 10
  Runner->>pr-review-report: Record skipped run and empty trace
  pr-review-report->>metrics/runs.jsonl: Append skip row
  refresh-human-queue->>metrics/runs.jsonl: Detect metrics change
  refresh-human-queue->>refresh-human-queue: Commit and push metrics ledger

Loading

Possibly related issues

• rainlanguage/rain-org-health issue 152 — The new producer-side skip rows provide the metrics data that this issue identifies as missing from the metrics page.

Possibly related PRs

• rainlanguage/issue-pr-cron#159 — Both changes update usage-gate handling in the runners and metrics command, including exit-code behavior.

Suggested labels: ai:ready

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes implement typed skip rows for both runners, preserve refusal behavior, publish metrics hourly, and avoid historical backfill for issue [#160].
Out of Scope Changes check	✅ Passed	The documentation, runner changes, metrics support, queue publishing, and tests all directly support the linked issue objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary change: recording usage-gate pauses as typed skip rows and publishing them hourly.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 2026-07-31-usage-gate-skip-rows

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@refresh-human-queue.sh`:
- Around line 120-127: Update refresh-human-queue.sh lines 120-127 to use git
diff --quiet HEAD -- for both snapshot_changed and metrics_changed probes, so
staged and unstaged changes are detected. In
pr-review-report-rs/tests/refresh_human_queue.rs lines 496-541, add a regression
case that stages the appended metrics/runs.jsonl skip row with git add before
the tick, then verifies the tick commits and pushes the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8ec950ff-0461-4890-bb5a-5a4252984d7f

📥 Commits

Reviewing files that changed from the base of the PR and between e69d4aa and 0ed4391.

📒 Files selected for processing (7)

• README.md
• campaign-run.sh
• pr-review-report-rs/src/main.rs
• pr-review-report-rs/tests/refresh_human_queue.rs
• pr-review-report-rs/tests/usage_gate_skip.rs
• refresh-human-queue.sh
• review-run.sh

[thedavidmeister]

Reviewed eb480fe: ready — typed skip rows (outcome:skipped, absent-not-null fields, refusal writes nothing) emitted through the one row-shape source, refresh-human-queue becomes the metrics committer awake mid-pause with either-file publish and snapshot-gated history, CodeRabbit's real index-vs-HEAD Major fixed TDD-style, four mutants killed, base-discrimination verified.