Publish the first measurement, and scope the README to it (TASK-12) - #20
Merged
Conversation
…ed against Nothing here said how well the tool works, so every claim about it was an opinion. This adds the first measurement and the ground it stands on. `truth.toml` fixes 31 defects of nextcloud/ios#4091 and 6 claims proven false. Each entry carries the evidence it was established by, so a later reader can disagree with a specific line rather than with the file. `origin` separates the 16 entries set by hand before any run from the 15 added by verifying a model's output against the code — a model graded on the second half is graded on ground it helped define, and the honest cross-model comparison uses the first. `docs/measurements.md` records eight series: three models, four checklist sizes, three reasoning and sampling settings. Recall, false positives, what the judge removes, and what each run cost, with the settings behind every number. The README carries two of those charts. The numbers are not flattering and are published anyway: 6% to 71% recall, and a false-positive rate whose lower bound is honest only because the unadjudicated findings are drawn as a range rather than assumed correct. They are a starting line to be moved, not a claim. Both charts count distinct defects rather than findings. Counting findings rewards a reviewer that writes "this file has no tests" thirty-six times.
The sample output was invented — a `feature/discount` branch, a `src/cart.py` that exists nowhere, token counts nobody measured. A reader takes that for a property of the tool. It is now a real run: muse-glimmer-30b over nextcloud/ios#4091, with the model and the merge request named, and with the note that four of the five findings shown are verified against the code and the fifth is not. The judge is no longer described as if surviving it meant being right. It removes 6% to 54% depending on how noisy the finder is, and what survives has a verified false-positive rate of its own, which the measurements page carries. "It does not replace a human reviewer" now says how much it misses instead of implying the gap is small, and a fourth entry states the thing most likely to get the tool switched off: it keeps no state between runs, so in CI it reports the same findings on every push, including the ones already read and dismissed. Nothing suppresses a repeat today, so the honest recommendation is a branch you run by hand.
Two things a reader was left to guess at. The scope: findings follow the change by consequence, not by authorship. Code the diff introduces is in scope, and so is code the diff breaks — including a call the branch now makes into something that was already wrong. The branch is what ships, and having been broken before this merge request does not make it work now. Problems the change does not touch stay out. The yardstick: this is not an attempt to reproduce a human review, and nothing in the measurements is an overlap-with-reviewer score. Agreement with a reviewer measures similarity, not correctness — a run that matched one perfectly would have added nothing, and the two miss different things. Every entry in the truth set was established against the code and carries the evidence for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing in the README said how well the reviewer works, so every claim about it
was an opinion — and the sample output was invented, down to a
src/cart.pythat exists nowhere. This adds the first measurement, the ground it stands on,
and rewrites the claims that no run had demonstrated.
What is here
truth.tomlfixes 31 defects of nextcloud/ios#4091and 6 claims proven false. Every entry carries the evidence it was established
by, so a reader can disagree with a specific line rather than with the file.
originseparates the 16 entries set by hand before any run from the 15 addedby verifying a model's output against the code.
docs/measurements.mdrecords eight series — three models, four checklistsizes, three reasoning and sampling settings — with recall, false positives,
what the judge removes, and what each run cost, plus the settings behind every
number.
README.mdcarries two of those charts, a real run in place of the inventedone, and four rewritten claims.
What the numbers say
Recall against the 31 known defects runs from 6% (nemotron-lightning, 8 items)
to 71% (Claude Opus 5). Verified false positives sit between 2% and 12%, drawn
as ranges because 2 to 15 findings per series remain unadjudicated. They are not
flattering and are published anyway: a starting line to be moved, not a claim.
Two conclusions worth a reviewer's attention. Model choice moved recall far more
than any knob — checklist size, reasoning depth and sampling all stayed inside a
few points. And the class the tool misses most is the one the reference pre-pass
already computes and prints into the prompt: a symbol that resolves nowhere.
Claims that were rewritten
6% to 54% depending on how noisy the finder is, and what survives has a
false-positive rate of its own.
same findings on every push — the usual reason such bots get switched off.
a call the branch now makes into already-broken code is in scope.
reviewer would measure similarity, not correctness.
What a reviewer should know
It is one merge request, so
n = 1at the repository level. Half the truth setis not independent: 15 of 31 entries were added by verifying one model's output,
so that model is graded partly on ground it defined — the honest cross-model
comparison uses the 16 marked
origin = "manual". Opus ran without the judge,as a ceiling rather than a configuration of this tool. And severity is not yet
stable enough to gate a pipeline on: the same defect came back
blockerin onerun and
minorin the next.The raw runs behind these numbers are not in the repository. If they should be,
say so and they can land as a separate change — trimmed, and without the config
files that have no business in an archive.
Closes TASK-12.