Skip to content

fix(eval): exclude unreadable PortSwigger outcomes from solve rates - #117

Merged
elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-ps-unreadable-oracle
Nov 16, 2026
Merged

elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-ps-unreadable-oracle

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

What does this change?

The automatic and parallel PortSwigger runners counted an expired or unreadable post-run lab as a failed solve. The manual runner already has an explicit ungraded state for the same condition. In a synthetic reproduction, a valid preflight and missing post-run widget produced a scored baseline failure.

Keep these records with solved=null and oracle_ok=false in both automated paths. Their cost remains visible, but the unreadable results fall outside the solve-rate denominator and cannot become parallel skills-retry candidates.

Validation

  • Reproduced the incorrect scored failure on main.
  • Exercised both runners with mocked pre- and post-run responses. Both preserve cost and leave the outcome unscored.
  • Offline eval harness passed. No launched labs or model calls.

@elementalsouls
elementalsouls merged commit a5d1710 into elementalsouls:main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants