feat(mutation): domain-specific operator packs via universalmutator (#141)#143
feat(mutation): domain-specific operator packs via universalmutator (#141)#143vincent-k2026 wants to merge 8 commits into
Conversation
First implementation step of issue #141: regex mutation packs that complement the cargo-mutants gate with the semantic faults it cannot express. Three packs, scoped to the files whose invariants they guard: - trie: level-base and bit-width arithmetic of the two-tier trie (STARTING_NODE_ID off-by-one, 8/24/40 bit-width confusion, fanout shift direction, ceil-rounding in the subtree-root climb) - canon: deletion/reversal of sort+dedup canonicalization on the proof path - statement deletion is not in cargo-mutants' repertoire - endian: to/from_le_bytes flips in the consensus hash and the 12-byte metadata codec (the issue #127 bug class) scripts/spec_mutation.sh generates mutants restricted to production lines (never inside #[cfg(test)]), then classifies each as invalid, killed, or survived; survivors fail the run and are reported as diffs. The Spec Mutation workflow runs it on demand (workflow_dispatch) and uploads the report. Local generation across all packs currently yields 59 mutants (50 trie, 3 canon, 6 endian). Families C/D/F from the issue are staged for after the first survivor triage; known gaps are documented in mutants/operators/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mutation testing - PASSNothing to test: no viable mutants were generated. |
Performance Benchmark ComparisonCompared Detailed Comparison
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a85c5d47f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
universalmutator exits 0 when a file yields no mutants and nonzero only for real invocation errors (missing source, unreadable rules or lines file), so tolerating its exit code could silently produce a partial manifest and a false PASS. Verified empirically against 1.14.1; the earlier local failure this guarded against was actually the since-fixed grep pipeline, not mutate. Raised by review on PR #143. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Address review feedback on PR #143: - endian.rules / trie.rules / canon.rules: anchor rules to call sites and add word boundaries so they can't match identifier substrings (`serialize_to_le_bytes`) or prefixes (`TRIE_WIDTH_BITS`); make the comparator-flip name-agnostic instead of hardcoding `a`/`b`. (Troublor) - spec_mutation.sh: drop comment-only mutants (universalmutator rewrites inside `///` docs → false SURVIVED), reported not silently discarded. (flyq #1) - spec_mutation.sh: cap lints to allow for the mutant build so a lint-only breakage (unused_mut under CI's -D warnings) isn't miscounted invalid. (flyq #2) - spec_mutation.sh: wrap nextest in a per-mutant timeout so a looping mutant is killed at 900s instead of hanging the job for hours. (flyq #3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c8889119a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Adopt the mega-evm spec-gate architecture per PR #143 review (Troublor): - Replace the regex packs + standalone spec_mutation.sh / spec-mutation.yml with the unified scripts/umutate.py driver (universalmutator --comby), scored by the SAME gate as cargo-mutants (scripts/mutation_gate.py) via the caught/missed/timeout/unviable results contract. - Express trie/canon/endian as comby structural packs under mutants/operators/<pack>/{manifest.toml,<pack>.rules}. Structural matching removes the regex-fragility class by construction: method/path-call templates can't hit an identifier substring, :[[word]] holes can't swallow a compound receiver; the one prefix-collision case (TRIE_WIDTH vs TRIE_WIDTH_BITS) is pinned with a boundary hole. - Fold everything into .github/workflows/mutation.yml: a diff-scoped blocking spec-gate on PRs (near-instant when no operator site changed), a nightly per-pack sweep, and an orphan-suppressions check that spans both engines. Verified locally with universalmutator 1.14.1 + comby 1.8.1: plan generates 57 mutants across the three packs (48 trie / 6 endian / 3 canon) with no identifier corruption and the TRIE_WIDTH boundary respected; a full endian run classified all 6 mutants as caught by the suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mutation testing - PASSNothing to test: no viable mutants were generated. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a879c9529
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Self-review fixes on the comby migration: - umutate.py: separate an unviable bucket. Comby operators produce genuinely non-compiling mutants (e.g. '<< BUCKET_SLOT_BITS ==> << BUCKET_ID_BITS' in a file that doesn't import BUCKET_ID_BITS), which the ported driver folded into 'caught' — inflating the score and regressing vs the original spec_mutation.sh invalid/killed split. Packs opt in with a 'build_cmd' (cargo check); a mutant that fails it is unviable and skips the test run. Verified: the BUCKET_ID_BITS mutant now classifies unviable while a viable '% (TRIE_WIDTH - 1)' mutant still compiles and is caught by a test. - mutation.yml: drop the redundant 'Warm test binaries' step from both spec-gate jobs (the lazy baseline/build already compile; it only wasted a full build on the common zero-mutant PR). cargo-mutants jobs keep theirs for copy_target seeding. Add a file-level header documenting the two engines + hygiene. - Reword stale 'spec-gate' phrasing (salt's packs are trie/canon/endian; the gate is the CI job) and document build_cmd in the operators README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58f305bbf9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
setup-rust-toolchain defaults RUSTFLAGS to "-D warnings". A valid, testable mutant can trip a warning — e.g. a canon sort-deletion leaves its `let mut` binding unused — which would fail the umutate build_cmd (`cargo check`) and be miscounted unviable instead of exercised. Set rustflags: "" on the spec-gate and spec-gate-sweep jobs so warnings stay non-fatal; genuine type/scope errors (the real unviable mutants) still fail. Verified: the entries.sort_unstable_by_key deletion now compiles and is caught by a test rather than dropped as unviable. (flyq #6 — mutants inside doc comments — needs no handling: comby is comment-aware and does not match tokens inside //, ///, //! by construction.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mutation.yml: pip -> pipx for universalmutator. ubuntu-24.04 runners block plain 'pip install' into the system env (PEP 668, runner-images#10897), which would fail the job before it produced results. - mutation.yml: add 'set -o pipefail' to the orphan-suppressions universe step. The default step shell lacks pipefail, so a failing 'umutate.py plan' piped into grep exited 0 and the orphan check would run on a partial universe. - endian pack: drop 'state/ahash/convert.rs' as a dead target. Its hash readers go through zerocopy::transmute! / .convert(), which these to/from_*_bytes rules can't match, so it generated zero mutants and overstated coverage. (The re-review also flagged unviable classification, already fixed in the prior commit via build_cmd.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e (flyq #143) Two config/doc inconsistencies from re-review, both resolved at the root in favour of the deliberate comby-only, line-independent design. 1. The manifest `mode` key was dead config. `generate_mutants` hardcodes `--comby` and nothing ever reads `pack["mode"]`; the comby migration (8a879c9) replaced the regex packs outright, so a `mode = "regex"` would silently do nothing. Remove the key from all three manifests and drop the `mode` / "regex mode" wording from the README (and a stale comment in umutate_setup.sh). `match` stays a regex file-content filter — that one is real and still used. 2. The triage policy advertised a `line = <n>` suppression pin that no schema field backs; `load_suppressions` ignores unknown keys and matching strips the locator on purpose. Implementing a line pin would fight the deliberate model (suppressions.toml matches by `(file, mutant-text)` precisely so they survive line shifts), so document the knobs that exist instead: a `kind = "line"` entry matched by locator-stripped text, and `kind = "function"` with a `pattern` regex. Verified: py_compile clean; load_packs loads all 3 packs with no `mode` key; `umutate.py plan --packs endian` still generates the 6 mutants; `mutation_gate.py report` suppresses a survivor by text regardless of line, and `orphans` matches across differing lines yet flags absent text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e (flyq #143) Two config/doc inconsistencies from re-review, both resolved at the root in favour of the deliberate comby-only, line-independent design. 1. The manifest `mode` key was dead config. `generate_mutants` hardcodes `--comby` and nothing ever reads `pack["mode"]`; the comby migration (8a879c9) replaced the regex packs outright, so a `mode = "regex"` would silently do nothing. Remove the key from all three manifests and drop the `mode` / "regex mode" wording from the README (and a stale comment in umutate_setup.sh). `match` stays a regex file-content filter — that one is real and still used. 2. The triage policy advertised a `line = <n>` suppression pin that no schema field backs; `load_suppressions` ignores unknown keys and matching strips the locator on purpose. Implementing a line pin would fight the deliberate model (suppressions.toml matches by `(file, mutant-text)` precisely so they survive line shifts), so document the knobs that exist instead: a `kind = "line"` entry matched by locator-stripped text, and `kind = "function"` with a `pattern` regex. Verified: py_compile clean; load_packs loads all 3 packs with no `mode` key; `umutate.py plan --packs endian` still generates the 6 mutants; `mutation_gate.py report` suppresses a survivor by text regardless of line, and `orphans` matches across differing lines yet flags absent text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
d548145 to
b04648e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b04648eae0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| targets = [ | ||
| "salt/src/proof/prover.rs", | ||
| "salt/src/proof/salt_witness.rs", | ||
| ] |
There was a problem hiding this comment.
When a PR changes create_sub_trie's bucket canonicalization, this pack won't run because salt/src/proof/subtrie.rs is not in the target list. I searched the proof path for sort/dedup sites and found the production bucket_ids.dedup() at salt/src/proof/subtrie.rs:227; the existing :[[recv]].dedup(); ==> rule would cover it if this file were targeted. Add proof/subtrie.rs so deleting that dedup cannot pass spec-gate with zero canon mutants.
Useful? React with 👍 / 👎.
| stderr=subprocess.DEVNULL).returncode == 0 | ||
|
|
||
|
|
||
| def baseline_ok(test_cmd: str) -> bool: |
There was a problem hiding this comment.
1. Baseline run can time out on a cold runner and fail the blocking gate with a misleading error
baseline_ok (scripts/umutate.py:277) runs the full nextest suite under the same 600s ANALYZE_TIMEOUT as mutants (umutate.py:51,257), and in CI that first run includes the cold build of this arkworks-heavy workspace. The repo already knows this build is long: both existing mutant jobs have a dedicated "Warm test binaries" step (mutation.yml:70,337) and .cargo/mutants.toml sets copy_target = true for the same reason. The new spec-gate (mutation.yml:144) and spec-gate-sweep (:254) jobs have no warm step, and rust-cache keys are per-job, so the first real run of each is cold. When the baseline exceeds 600s, ensure_baseline (umutate.py:358-366) exits with "baseline failed … fix the failing tests" — wrong diagnosis, flaky blocking gate. Suggested fix: add the same warm step (cargo nextest run … --no-run) to both jobs, and in umutate.py give the baseline either no timeout or a separate, larger one, with an error message that distinguishes timeout from failure.
| def write(name, items): | ||
| (out / name).write_text("\n".join(items) + ("\n" if items else "")) | ||
|
|
||
| write("caught.txt", caught) |
There was a problem hiding this comment.
2. The nightly sweep loses all results on a job timeout.
cmd_run writes caught.txt/missed.txt/etc. only after every mutant is analyzed (umutate.py:389). The trie pack is ~50 mutants analyzed sequentially with a 330-minute job limit (mutation.yml:258); if the limit hits, the if: always() artifact upload finds an empty directory and hours of compute are discarded. Appending to the result files as each mutant is classified (or flushing per source file) would make partial runs useful.
3 participants
Summary
First implementation step of #141 — the semantic mutation layer complementing the
cargo-mutantsgate:mutants/operators/{trie,canon,endian}.rules— families A, E and B from the issue, scoped per pack (the selection filter: what soundness or determinism invariant would a surviving mutant violate?). Generic operators are deliberately excluded; thecargo-mutantsgate owns those.scripts/spec_mutation.sh— generates mutants with universalmutator (--onlyour rules,--linesrestricted to production code so mutants never land in#[cfg(test)]), then classifies each as invalid / killed / survived. Survivors fail the run and are reported as diffs.Spec Mutationworkflow —workflow_dispatchwith apacksinput; uploadsreport.md+survivors.txtand writes the step summary.Validated locally
Generation across all three packs yields 59 mutants (50 trie, 3 canon, 6 endian), each confirmed to be exactly the intended semantic fault (spot-checked diffs:
STARTING_NODE_ID[level+1]→[level],>>MIN_BUCKET_SIZE_BITS→>>(MIN_BUCKET_SIZE_BITS+1),MAX_SUBTREE_LEVELS-1→-2, sort/dedup deletion,le↔beflips). The kill-analysis loop needs cargo and runs in CI (~a few hours for all packs; killed mutants stop at the first failing test).Notes
endianpack finds nothing inconvert.rsuntil fix(hasher): make the deterministic hasher endian-independent #142 lands (it currently usestransmute); after that it guards the new explicit-LE reads.gh workflow run "Spec Mutation"and triage the survivors — each is either a missing test or a documented equivalence.Refs #141.
🤖 Generated with Claude Code