Skip to content

fix(eval): grade PortSwigger labs from the actual status widget - #113

Merged
elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-ps-oracle-widget
Nov 16, 2026
Merged

elementalsouls merged 1 commit into
elementalsouls:mainfrom
rudycelekli:rudy/cbh-ps-oracle-widget

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

What does this change?

The PortSwigger evaluator treated any "Solved" text or class anywhere in the page as proof of success. On a synthetic non-lab response, an unrelated Solved label was graded as a solved lab.

Read status only from class tokens on the lab-status widget. This handles class order and extra classes, ignores unrelated page text, and leaves a conflicting or missing widget ungraded.

Validation

  • Reproduced the incorrect solved verdict on main.
  • Oracle self-test passed all eight cases, including reordered classes, an unrelated label, inline JavaScript and conflicting widgets.
  • Existing offline eval harness passed. No hosted lab or model calls.

@elementalsouls
elementalsouls merged commit d4472b1 into elementalsouls:main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants