Skip to content

Publish the first measurement, and scope the README to it (TASK-12) - #20

Merged
axazeano merged 4 commits into
mainfrom
feature/TASK-12-measurements-and-truth
Aug 20, 2026
Merged

Publish the first measurement, and scope the README to it (TASK-12)#20
axazeano merged 4 commits into
mainfrom
feature/TASK-12-measurements-and-truth

Conversation

@axazeano

Copy link
Copy Markdown
Owner

Nothing in the README said how well the reviewer works, so every claim about it
was an opinion — and the sample output was invented, down to a src/cart.py
that exists nowhere. This adds the first measurement, the ground it stands on,
and rewrites the claims that no run had demonstrated.

What is here

truth.toml fixes 31 defects of nextcloud/ios#4091
and 6 claims proven false. Every entry carries the evidence it was established
by, so a reader can disagree with a specific line rather than with the file.
origin separates the 16 entries set by hand before any run from the 15 added
by verifying a model's output against the code.

docs/measurements.md records eight series — three models, four checklist
sizes, three reasoning and sampling settings — with recall, false positives,
what the judge removes, and what each run cost, plus the settings behind every
number.

README.md carries two of those charts, a real run in place of the invented
one, and four rewritten claims.

What the numbers say

Recall against the 31 known defects runs from 6% (nemotron-lightning, 8 items)
to 71% (Claude Opus 5). Verified false positives sit between 2% and 12%, drawn
as ranges because 2 to 15 findings per series remain unadjudicated. They are not
flattering and are published anyway: a starting line to be moved, not a claim.

Two conclusions worth a reviewer's attention. Model choice moved recall far more
than any knob — checklist size, reasoning depth and sampling all stayed inside a
few points. And the class the tool misses most is the one the reference pre-pass
already computes and prints into the prompt: a symbol that resolves nowhere.

Claims that were rewritten

  • The judge was described as though surviving it meant being right. It removes
    6% to 54% depending on how noisy the finder is, and what survives has a
    false-positive rate of its own.
  • "Does not replace a human reviewer" now states the size of the gap.
  • A new entry says the tool keeps no state between runs, so in CI it repeats the
    same findings on every push — the usual reason such bots get switched off.
  • Scope is stated: findings follow the change by consequence, not authorship, so
    a call the branch now makes into already-broken code is in scope.
  • Reproducing a human review is explicitly not the goal — agreement with a
    reviewer would measure similarity, not correctness.

What a reviewer should know

It is one merge request, so n = 1 at the repository level. Half the truth set
is not independent: 15 of 31 entries were added by verifying one model's output,
so that model is graded partly on ground it defined — the honest cross-model
comparison uses the 16 marked origin = "manual". Opus ran without the judge,
as a ceiling rather than a configuration of this tool. And severity is not yet
stable enough to gate a pipeline on: the same defect came back blocker in one
run and minor in the next.

The raw runs behind these numbers are not in the repository. If they should be,
say so and they can land as a separate change — trimmed, and without the config
files that have no business in an archive.

Closes TASK-12.

…ed against

Nothing here said how well the tool works, so every claim about it was an
opinion. This adds the first measurement and the ground it stands on.

`truth.toml` fixes 31 defects of nextcloud/ios#4091 and 6 claims proven false.
Each entry carries the evidence it was established by, so a later reader can
disagree with a specific line rather than with the file. `origin` separates the
16 entries set by hand before any run from the 15 added by verifying a model's
output against the code — a model graded on the second half is graded on ground
it helped define, and the honest cross-model comparison uses the first.

`docs/measurements.md` records eight series: three models, four checklist sizes,
three reasoning and sampling settings. Recall, false positives, what the judge
removes, and what each run cost, with the settings behind every number.

The README carries two of those charts. The numbers are not flattering and are
published anyway: 6% to 71% recall, and a false-positive rate whose lower bound
is honest only because the unadjudicated findings are drawn as a range rather
than assumed correct. They are a starting line to be moved, not a claim.

Both charts count distinct defects rather than findings. Counting findings
rewards a reviewer that writes "this file has no tests" thirty-six times.
The sample output was invented — a `feature/discount` branch, a `src/cart.py`
that exists nowhere, token counts nobody measured. A reader takes that for a
property of the tool. It is now a real run: muse-glimmer-30b over
nextcloud/ios#4091, with the model and the merge request named, and with the
note that four of the five findings shown are verified against the code and the
fifth is not.

The judge is no longer described as if surviving it meant being right. It
removes 6% to 54% depending on how noisy the finder is, and what survives has a
verified false-positive rate of its own, which the measurements page carries.

"It does not replace a human reviewer" now says how much it misses instead of
implying the gap is small, and a fourth entry states the thing most likely to
get the tool switched off: it keeps no state between runs, so in CI it reports
the same findings on every push, including the ones already read and dismissed.
Nothing suppresses a repeat today, so the honest recommendation is a branch you
run by hand.
Two things a reader was left to guess at.

The scope: findings follow the change by consequence, not by authorship. Code
the diff introduces is in scope, and so is code the diff breaks — including a
call the branch now makes into something that was already wrong. The branch is
what ships, and having been broken before this merge request does not make it
work now. Problems the change does not touch stay out.

The yardstick: this is not an attempt to reproduce a human review, and nothing
in the measurements is an overlap-with-reviewer score. Agreement with a reviewer
measures similarity, not correctness — a run that matched one perfectly would
have added nothing, and the two miss different things. Every entry in the truth
set was established against the code and carries the evidence for it.
@axazeano
axazeano merged commit 5de4171 into main Aug 20, 2026
4 checks passed
@axazeano
axazeano deleted the feature/TASK-12-measurements-and-truth branch August 26, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant