Skip to content

fix(eval): require the explicit final line for false-positive grading - #116

Merged
elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-fp-verdict-contract
Nov 16, 2026
Merged

elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-fp-verdict-contract

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

What does this change?

The false-positive evaluator guesses a binary verdict from prose whenever the required final line is missing. Main grades "No hypothesis was confirmed" as VULNERABLE, and scores a quoted example verdict even when the response makes no decision.

Parse the explicit final verdict line requested by the evaluator's prompt. Empty output is reported as NO_OUTPUT, and other malformed responses stay INCONCLUSIVE.

Validation

  • Reproduced both incorrect classifications on main.
  • Added seven offline parser cases to the existing eval harness, including valid final-line controls. The harness passes.
  • No model calls or live targets.

@elementalsouls
elementalsouls merged commit ce61d87 into elementalsouls:main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants