Skip to content

fix(eval): report false-positive rates alongside grading coverage - #131

Merged
elementalsouls merged 2 commits into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-fp-coverage-metrics
Nov 16, 2026
Merged

elementalsouls merged 2 commits into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-fp-coverage-metrics

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

What does this change?

The false-positive summary counts empty or inconclusive responses in the denominator as if they were graded. With one false positive and one unanswered safe case it reports 50% FPR, although the one graded case was a false positive. A condition that frequently fails to return a verdict can appear safer.

Report rates over scored responses and print safe and vulnerable grading coverage beside each condition. An empty scored denominator is displayed as n/a.

Validation

  • Offline main-path regression uses a safe false positive, an unanswered safe case and an unanswered vulnerable control. It verifies 1/1 FPR, unavailable TPR, and 1/2 safe and 0/1 vulnerable coverage.
  • Existing eval harness passes. No model calls or external targets.

@elementalsouls
elementalsouls merged commit 2f9a226 into elementalsouls:main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants