Skip to content

fix(eval): leave unreadable Juice Shop outcomes ungraded - #141

Merged
elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-juice-oracle-loss
Nov 16, 2026
Merged

elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-juice-oracle-loss

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

An unreadable post-run Juice Shop oracle was recorded as a miss, while a string-valued solved flag could count as a solve. Require a Boolean oracle verdict, retain the attempted run and its cost, and exclude unreadable outcomes from solve denominators.

Verification

Offline eval discovery and harness sanity. The regression fails on main for missing, malformed and non-Boolean oracle data.

@elementalsouls
elementalsouls merged commit f74dd51 into elementalsouls:main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants