review: the live A/B runner (phase 3)

[jwbron]

Phase 3 of the live A/B eval plan (#232), stacked on the Phase 2 producer (#234): score live runs against labeled ground truth and diff two review.md versions arm to arm.

3a: eval/live-match.ts

Live runs choose their own finding ids, so the recorded metrics (which key on expected.mustCatch ids) cannot score them. The matcher maps a run's POSTED candidates onto the case's labeled defect specs: a candidate satisfies a spec when its anchor agrees with the spec's path (and line window, when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose, case-insensitively. Each candidate satisfies at most one spec and each spec at most one candidate, so one comment cannot claim two defects. An injected fallback arbiter (hard-capped, same-file candidates only, recorded as via: "fallback" for human audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates the live analogues of the suite's numbers: must-catch recall, verdict agreement, clean false-flag (a clean case that blocks counts), and noise (posted candidates matching no spec).

3b: eval/live-ab.ts

The arm orchestrator and CLI. Baseline review.md comes from git show <merge-base> (or --base-ref), candidate from the working tree; both arms run over the same live corpus with everything else (corpus, lib/, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model-behavior seam. Each arm runs under half the --max-usd budget (default 40 total) with sticky exhaustion: once spend plus the running per-case average crosses the cap, the remaining cases are recorded as SKIPPED and the report still emits (dying at a cap with nothing posted is the failure mode the plan forbids). Each live result replays through the deterministic pipeline; runner.ts gains an optional RunOptions.validation override so the live validator's output replaces the recorded block. Spec-level regressions ("baseline caught, candidate missed" and the reverse) are diffed only over cases both arms actually scored, so a budget skip is never reported as a regression.

Judge scoring reuses the pinned judge for quality aggregates only (judge-vs-ground-truth disagreement keys on recorded ids, which live arms do not use); the fetch model moves from live-judge.ts into shared eval/judge-live-model.ts.

Output: out/live-ab-report.json plus a markdown table (metrics deltas, judge quality delta, cost, wall clock, regressions, skips, agent failures) printed and appended to GITHUB_STEP_SUMMARY. Report-only, with one exception per the playbook's standing rule: the candidate arm failing any adversarial-injection case (wrong verdict or missed spec) exits non-zero.

Test plan:

• pnpm run test --run: 696 tests green (16 new: matcher rules incl. window overlap, one-candidate-one-spec, malformed-regex fallback-to-literal, capped fallback audit trail; runArm budget semantics incl. the sticky-stop case that caught a real bug in review; regression diffing over shared cases; report rendering both gate outcomes).
• pnpm run typecheck and eslint clean.
• The end-to-end CLI (pnpm dlx tsx workflows/review/eval/live-ab.ts --max-usd 10 --cases incident-money-rounding,clean-no-findings) needs ANTHROPIC_API_KEY; this environment has none, so that is the first thing to exercise when testing. On an unchanged review.md it prints the identical-arms note and near-zero deltas.

Next steps (human)

1. End-to-end A/B validation DONE, superseded by the phase 4 acceptance runs (linked on review: per-PR live A/B workflow (phase 4) #237): the control arm held recall and verdict agreement flat at +0% and the weakened arm showed -17% recall with the lost spec named, which is also the first empirical read on matcher quality (7 cases matched; one systematic miss traced to lens routing, fixed on review: live-enabled corpus format and ten live cases #233, not to the matcher). Two acceptance-driven additions landed here since: perCase.failedAgents entries now carry <agent>: <reason> (a claim-validator flake was undiagnosable from the report alone; the flake itself is still to be root-caused on a future run), and every report closes with a per-row stability footer (judge quality rose on the deliberately regressed arm because fewer, surer comments each score better, so the footer warns against reading it alone).
2. Review the two scoring caveats: judge output is quality-aggregates only (disagreement semantics key on recorded ids), and the fallback arbiter is wired but not yet given a live implementation (matches would be marked via: "fallback" for audit when one is added).
3. Confirm the budget semantics (half of --max-usd per arm, sticky skip-and-report) match how you want cost bounded.

---

Update 2026-07-09: eval-tuning pass (three riders)

Three instrument fixes from the eval-tuning memo, landed here so the whole stack inherits them:

1. Pre-flight identity short-circuit (memo item 1, hoisted down from review: re-review coverage for the eval corpus (open-PR cases + lifecycle chain) #251 with identical text so that branch merges cleanly): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; --force-arms preserved for wobble controls. Corpus-only and plumbing-only PRs in this stack now pay $0 per push.
2. Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake (the review: re-review accountability, code-rendered from the reconciler keep-list #244/review: hand off out-of-lane observations instead of dropping them #245 incident shape). Retry spend is recorded in the report.
3. Drop-bucket taxonomy (memo item 4): every missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/validation) by re-matching against the dropped candidates with the line window relaxed. Lost regressions are annotated with their class and the table gains a found-but-dropped count row.

Plus judge economics: the judge pin moves from Opus 4.8 to claude-haiku-4-5-20251001 (the judge grades prose only; every load-bearing metric is deterministic, and the acceptance runs showed judge quality is not single-run-stable on any model, so it is priced accordingly; supersedes operator direction 4), and the judge seam gains retry with backoff so a transient 500 no longer wastes a whole scoring pass.

[changeset-bot]

🦋 Changeset detected

Latest commit: 193ae69

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jwbron]

Two follow-ups from the phase 4 acceptance runs (details in the plan doc on #232):

1. Persist agent-failure reasons. claim-validator failed on incident-cache-missing-key in 2 of 4 arm-runs across the two acceptance runs (flaky; the case still caught its defect and verdict, so degradation is graceful). The report and job log carry only the agent name, so the failure is undiagnosable without a rerun. The runner should write the reason (parse error, timeout, turn cap) into live-ab-report.json per failed agent.

2. State per-row stability in the report footer. On the no-op control arm, judge mean quality moved -0.11 and noise +5%; at N=7 those rows are single-run jitter. A footer naming which rows are stable (recall, verdict agreement, regressions, adversarial gate) and which need repeated runs (judge quality, noise) keeps readers from chasing noise.