review: add the live A/B eval plan

[jwbron]

Adds workflows/review/eval/live-ab-plan.md: a phased, agent-executable plan for real-model evals of review-bot changes, making the round-two doc's eval playbook executable for its behavior-shifting and cost-affecting change classes (today only the deterministic replay exists, which is structurally blind to prompt and model changes).

The phases:

1. Live-enabled corpus cases: real diffs, post-change file trees, PR context, and labeled defect specs (live runs choose their own finding ids, so ground truth matches on anchor window + mechanism, not id). Implementation is up in a separate PR.
2. Live producer: run the actual sub-agent roster from review.md over a case via the Agent SDK (Read/Grep/Glob only, staged checkout, per-agent model pinning, live claim-validator, cost capture), behind the produceFindings seam runner.ts already ships.
3. A/B runner + delta report: same-run arms (merge-base review.md vs working tree), five-metric deltas, judge quality deltas, per-case cost, adversarial hard gate outright.
4. CI wiring: review-eval-ab.yml on paths: [workflows/review/**], sticky PR comment, smoke-live subset by default, full-eval label escalation.

Settled decisions recorded in the doc: same-run A/B over stored baselines; candidate corpus + lib for both arms (isolate the model seam); bounded agentic sub-agents, not one-shot completions; report-only except the adversarial gate; budget caps that still emit a partial report. Also records how this instrument relates to seeded live-PR trials like Khan/webapp#40678 (complementary layers, different cadences).

Test plan:

Docs only (plus an empty changeset). pnpm run test --run stays green.

Next steps (human)

1. Review the settled design decisions section; they bind the whole stack (review: live-enabled corpus format and ten live cases #233-review: add the review-trial skill (live A/B phase 5) #238), so disagreement is cheapest to voice here.
2. Two decisions the plan deliberately leaves to humans, no code blocked on them: ratify the gate thresholds that would turn the A/B report into a merge gate (until then everything is report-only except the adversarial rule), and decide whether pattern-triage joins the live roster.
3. This PR is docs plus an empty changeset; it can merge independently of the stack whenever the plan reads right. All five phases have landed and the doc now records the phase 4 acceptance results (both arms passed; the plan's acceptance section names the runs and the judge-direction lesson), so the remaining human items are exactly the two decisions in step 2 plus the consumer-repo trial that accepts review: add the review-trial skill (live A/B phase 5) #238.

[changeset-bot]

🦋 Changeset detected

Latest commit: 906a1fb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jwbron]

Heads up: the eval chain (#233 -> #234 -> #236 -> #237 -> #238, plus #235) was cut from a main that predated review@1.4.0 (missing #228/#230), so its review.md lacked the v1.4.0 disciplines. All chain branches are now refreshed with main via bottom-up merges (no history rewrites), and both acceptance arms (#239/#240) got the refreshed base merged into their heads so the noise-floor and recall-signal measurements characterize the v1.4.0 reviewer that actually runs in production. #240's weakening edit was verified to still isolate to the correctness reviewer against the v1.4.0 text. The refresh pushes will re-trigger the Review Eval A/B runs on both arms; treat the new reports as the acceptance record.