From 8cb01658f14f498d8afa5e7bc4363d50c049068b Mon Sep 17 00:00:00 2001 From: baku-ccron Date: Tue, 18 Aug 2026 13:57:18 +0000 Subject: [PATCH 1/3] =?UTF-8?q?docs(qa-guide):=20=C2=A73=20routes=20mutati?= =?UTF-8?q?on=20validation=20to=20mutation-probe,=20not=20a=20hand=20loop?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #336 §3 was a complete self-contained recipe — "apply ONE targeted mutation … confirm the test fails, restore" — and never named `mutation-probe`, the bin the `adversarial-mutation-test` skill runs at its step 3. campaign-prompt.txt step 4 already says to INVOKE the skill rather than hand-copy its rules, but §3 wins for anyone reading the guide alone, and §8's mandatory `Mutations applied` line sends every producer back to it. So agents wrote their own harness every run. The tool is now the route, with the runnable command, `--help` as the manual for file format / verdicts / exit codes, and the four integrity properties stated as the REASON: green non-empty baseline, proof from the suite's own tally that it RAN (NO-RUN is not a pass), an exactly-once target, and a byte-exact restore re-checked before the next mutant. A hand loop can assert all four and prove none. The two duties the bin cannot discharge — commit before the first probe, keep targets out of test code — are named as the producer's. §8's `Mutations applied` line now says its content is TRANSCRIBED from the probe's verdicts, and step 3c(ii)'s "(break the covered line → the assertion FAILS → restore → PASSES)" points at the same probe. The hand-roll is a prohibition, not an omission: a rule merely deleted is a rule the next edit reinvents, which is why the retirement is pinned by a test the way `the_producer_prompt_routes_screenshots_to_this_subcommand` pins #251's. The guide is charged to the prompt cap because campaign-prompt.txt names it, and it grows 1,077 bytes net of step 3c(ii)'s cut. Cap tracks the measurement: 154969. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/rust.yml | 2 +- QA-GUIDE.md | 29 +++++++++--- campaign-prompt.txt | 2 +- pr-review-report-rs/src/main.rs | 78 +++++++++++++++++++++++++++++++++ 4 files changed, 104 insertions(+), 7 deletions(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index e907811b..bd23e41c 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -443,7 +443,7 @@ jobs: - uses: rainlanguage/rainix/.github/actions/prompt-cap@main with: paths: "**/*prompt*" - cap: 153919 + cap: 154969 # The plugin prompts, capped separately from the cron runners above. Same tax, # different payer: a `/nr` or `/ncc` dispatch reads its agent file whole into a # fresh window, so every byte is charged once per turn of that agent's run, and diff --git a/QA-GUIDE.md b/QA-GUIDE.md index 2c1d2a9a..17960664 100644 --- a/QA-GUIDE.md +++ b/QA-GUIDE.md @@ -47,11 +47,28 @@ Every behavior the diff claims to fix or add gets a test that: sides of your change has pinned nothing (cyclo.site#398: three deploy-gate fixes, 15KB test file untouched, every test green before AND after). -## 3. Mutation-validate the new tests +## 3. Mutation-validate the new tests — with the bundled tool -For each new test, apply ONE targeted mutation to the line it claims to cover -(negate the guard, flip the comparison, drop the call), confirm the test fails, -restore. A test that survives its own mutation is decoration. +The route is the `adversarial-mutation-test` skill scoped to your change, whose +probe step authors ONE targeted mutation per behavior (its catalog) into a +`mutants.toml` and runs the bundled bin: + +```sh +nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.toml +``` + +`mutation-probe --help` is the manual: file format, verdicts, exit codes. Do +NOT hand-roll an edit-run-restore loop — the bin ENFORCES what a hand loop can +only assert. A red, silent or zero-test baseline aborts before any probe; the +suite's own tally proves it RAN, so a crash or compile error is NO-RUN and +never a pass; each target must occur EXACTLY once in its file; and every +restore is verified byte-exact before the next mutant. Two duties stay yours: +COMMIT before the first probe, and keep targets out of test code — a mutant in +the oracle co-mutates the expectation and voids the result. + +SURVIVED is a real gap: strengthen the test in place or add one, re-probe +(`--only`) until KILLED, and never edit a test to pass under a mutation. A test +that survives its own mutation is decoration. ## 4. Oracle discipline @@ -101,7 +118,9 @@ don't hold. All four lines are required. A line your change cannot have takes `n/a` **with the reason** (a docs-only diff has no mutations to apply); an absent line is not -an option. +an option. `Mutations applied` is TRANSCRIBED from the probe's verdicts (§3) — +the mutant, and the test that KILLED it — never recalled; a mutation nothing +scored is not evidence. That is enforced where the PR is opened, not only where it is judged. You open PRs with the `open_pr` MCP tool, which reads the body file and REFUSES (exit 3) diff --git a/campaign-prompt.txt b/campaign-prompt.txt index 5a0d9c32..e451be05 100644 --- a/campaign-prompt.txt +++ b/campaign-prompt.txt @@ -51,7 +51,7 @@ Each run: • PRE-EXISTING `main` RED (the failing check ALSO fails on the base branch, not introduced by this PR) is FIXABLE and is NOT a "leave it": a branch CAN carry ANY fix, INCLUDING the root-cause fix for a red `main`. Do NOT paper over it — a merge-main-only commit or an empty-commit retrigger CANNOT green a genuinely-broken test, and re-doing them every run is the churn this rule exists to kill (a PR with 4+ merge-main / empty-retrigger commits still red is the smell). Instead: READ the failing log, diagnose the ACTUAL failure on `main`, and FIX IT ON THE BRANCH (mutation-validated per (ii) if it touches a test — a real fix, NEVER a weakening) so the check goes green on the PR. Because the same breakage sits on `main`, ALSO open a dedicated minimal fix-`main` PR (one issue per PR) carrying that same root-cause fix — one such fix typically greens the whole cohort of PRs branched off that red `main`. • CANNOT-RESOLVE → FLICK TO DESIGN, never re-churn. If after a genuine diagnosis the red cannot be greened on the branch because doing so needs a HUMAN DECISION — the failing assertion encodes an ambiguous spec / design question, or the only available "fix" is a judgment call the producer must not make unilaterally (weakening an assertion, choosing between incompatible behaviours, a cross-repo contract change) — do NOT leave it to re-attempt every run. FLICK IT TO DESIGN via the transition: `pr-review-report flag-design ""` — the reason IS the question. A design-flagged PR is PARKED (its `ai:design` label surfaces it in `human-queue` and is the cross-run marker; skip it like any human-gated state, never re-attempt). Never claim a red fixed without reading its log. - (ii) Clone fresh into `{{WORK_DIR}}/-pr`, `gh pr checkout ` (if that dir already exists from a prior run, RE-SYNC to the PR branch with checkout-force + clean — `git -C fetch origin && git -C checkout -f -B origin/ && git -C clean -fdx` — NEVER `git reset --hard`, which is forbidden and hook-blocked). REPRODUCE the red locally on the UNCHANGED checkout FIRST: if the failing check is already GREEN locally before you change anything (you cannot reproduce it), it is CI-only — do NOT push a no-op code "fix"; handle it per (i) (empty-commit re-trigger if the log shows a transient flake, else LEAVE and note). Only once you reproduce the red: make the MINIMAL fix for exactly that failure. If the fix touches a test/assertion you MUST prove it is a correctness fix, NOT a weakening: mutation-validate per step 4 (break the covered line → the assertion FAILS → restore → PASSES); making the test pass by loosening/deleting an assertion is FORBIDDEN. A lint/format failure → run the repo's formatter. Then build + run the failing check locally to CONFIRM it goes green. + (ii) Clone fresh into `{{WORK_DIR}}/-pr`, `gh pr checkout ` (if that dir already exists from a prior run, RE-SYNC to the PR branch with checkout-force + clean — `git -C fetch origin && git -C checkout -f -B origin/ && git -C clean -fdx` — NEVER `git reset --hard`, which is forbidden and hook-blocked). REPRODUCE the red locally on the UNCHANGED checkout FIRST: if the failing check is already GREEN locally before you change anything (you cannot reproduce it), it is CI-only — do NOT push a no-op code "fix"; handle it per (i) (empty-commit re-trigger if the log shows a transient flake, else LEAVE and note). Only once you reproduce the red: make the MINIMAL fix for exactly that failure. If the fix touches a test/assertion you MUST prove it is a correctness fix, NOT a weakening: mutation-validate per step 4 (the skill's `mutation-probe` pass, never a hand loop); making the test pass by loosening/deleting an assertion is FORBIDDEN. A lint/format failure → run the repo's formatter. Then build + run the failing check locally to CONFIRM it goes green. (iii) Push the fix as a NEW fast-forward commit on top of the branch tip — with the `mcp__fsm__push` TOOL, `{"clone": ""}` (add `"branch": ""` only when the local branch is named something else). PUSHING IS A TOOL, never a bare `git push`: the tool's result is this run's TYPED RECORD that the rework happened — it reports the sha it created and the PR whose head that sha is, which is what `work-tokens` counts as a work item, and a `git push` in Bash records nothing any reader can join to a PR. It cannot spell a force-push at all. NEVER force-push in ANY form or spelling: not `--force`, `-f`, `--force-with-lease`, `--force-if-includes`, a leading-`+` refspec (`git push origin +branch`), a `HEAD:branch --force`, NOR any of these routed through `git -C push …`. If a plain push is rejected as non-fast-forward, STOP and leave the PR for a human — never reach for a force/lease/refspec variant to land it. Never `git reset --hard` (any form, incl. `git -C `), never rebase-force, never change the PR's scope or its linked issue. Only ever fix PRs authored by `{{ASSIGNEE}}`; never touch another author's PR or branch. (iv) MIGRATION — the deploy-shaped case, triggered by EITHER (a) a red prod-pin / `testProdDeploy*` / deploy-pinned-constant check (REPLACES (ii)/(iii) for it: there is no code fix to push), OR (b) a GREEN PR carrying a legacy `REQUIRES redeploy at land` marker in its body or a trusted note. Under the split deploy/abstract release lifecycle NO merge waits on a deploy (#162): deploy repos freeze per-tag records and publish on `rainix-tag-release`, the on-chain deploy is a separate manual workflow dispatch decoupled from publishing, and library repos never deploy — so either trigger firing is evidence of exactly ONE thing: THIS REPO HAS NOT MIGRATED to the split lifecycle. The PR is blocked on that migration, not on any deploy, and you deploy NOTHING, EVER (there is no producer deploy path; deploys are a human's decoupled manual dispatch). Route it: (1) find the repo's lifecycle-migration issue/PR (search the repo's open issues/PRs for the split-lifecycle / release-workflow migration); (2) if none exists, FILE it — `gh issue create` in that repo, assignee {{ASSIGNEE}}, shaped as the audit skill's lifecycle-split finding (the repo keeps a single "current" deployment pin that premerge CI tests against, so every bytecode-changing PR goes red until an out-of-band deploy; the fix is the split lifecycle: per-tag frozen deploy records + `rainix-tag-release` publishing, with the on-chain deploy a decoupled manual dispatch) — this is the ONE sanctioned `gh issue create` and the only exception to step 4's no-new-issues posture; (3) TRANSITION the PR: `pr-review-report flag-blocked-on "repo not migrated to the split release lifecycle: " --blocked-by ` — the typed dep makes clearance automatic: the vetter clears the flag the run after the migration lands and the PR re-enters vetting fresh. An APPROVED PR is human-sacred and cannot be flagged — report it as blocked-on-migration in the run summary instead (still do (1)/(2) so the dep exists). DEDUP: if the PR already carries `ai:blocked-on` naming the migration, it is parked — do not re-flag. There is no blocked-deploy state and no producer deploy any more (#162/#221) — a PR still wearing the string is an unmodelled label, so treat it as any other PR and route it by the rules above. Still NEVER merge; landing stays interactive on the human's explicit word. 3c. BACKFILL MISSING SCREENSHOTS on your OWN open UI PRs (still existing-work — after 3b's red fixes, before step 4 opens anything new): a UI PR with no screenshot is, like a red PR, in-flight work that is NOT review-ready (step 5), so finish it before starting new work. YOUR LIST IS STEP 2a's `worklist` ROWS WHOSE `nextAction` IS `screenshot-3c`, never a path list written here: `worklist`'s `is_ui_path` is the ONE definition of which paths fall under this requirement — raindex's `packages/webapp` and `packages/ui-components`, everything under a `site/` dashboard tree INCLUDING the `health.json` its panels draw, and every `.svelte`/`.css`/`.html` file wherever a repo keeps it (`cyclo.site` keeps its components in `src/lib/components/`, which is where both of #140's PRs lived) — and a second enumeration in this step is a second definition, free to drift from it while reading as though it agrees. The row's `markers.uiTouch` tells you which answer routed it: `yes` a path under the requirement is in the changed list, `unknown` the changed list could not be ruled out — BOTH route here, because `unknown` means nothing ruled UI out, not that there is none. For each, judge FROM THE DIFF whether it changes RENDERED OUTPUT a user sees (per step 5: `.svelte` markup, a visible component/element, user-visible styling, a non-`.svelte` service whose output is directly displayed — chart data / labels / formatters — OR a `site/` dashboard page's visible markup/section). A change that ONLY touches `