Skip to content

ci(eval): harness sanity gate + baseline scaffold - #67

Merged
elementalsouls merged 1 commit into
mainfrom
eval/harness-sanity
Aug 24, 2026
Merged

elementalsouls merged 1 commit into
mainfrom
eval/harness-sanity

Conversation

@elementalsouls

Copy link
Copy Markdown
Owner

Phase 3 (BugHunter) — the CI-able half of capability eval.

The full ablation eval (run_eval*.py — skills-on vs skills-off on a self-grading target) needs Docker + Burp + an authed claude CLI, so it runs locally, not on a hosted runner. This ships the part CI can do — keeping the harness from rotting — plus a place to record the real numbers.

What this adds

  • .github/workflows/eval-harness.yml — on eval/ changes: py_compile every eval/*.py, run the oracle parser self-test, run the new harness sanity test. Network-free, contents: read.
  • eval/test_harness.py — asserts (1) every eval/*.json config parses with the keys the runners expect, (2) oracle_portswigger classifies the lab-status widget correctly, (3) fp_app.py starts and serves every path in fp_cases.json (keeps the false-positive trap app and its ground-truth cases in sync). Stdlib, localhost only.
  • eval/BASELINE.md — the local run procedure + a results table to fill in (skills-on/off solve-rate delta) and commit after a real run.

What's NOT here

The actual skills-on/off baseline numbers — those require your local Docker/Burp/authed-CLI setup. Run eval/run_eval.py (Juice Shop v0) or eval/run_eval_ps_auto.py (PortSwigger v1), then fill in and commit eval/BASELINE.md.

Note

Building test_harness.py caught two bugs in my own probe (not the app): appending a canary to an exact-match route corrupted it, and urllib auto-followed the open-redirect trap's 302 to a 404. Fixed (query-aware URL + no-follow opener); fp_app and fp_cases are confirmed in sync.

Verified

py_compile all eval scripts · oracle self-test 4/4 · harness test PASS · leak-guard 0 matches.

🤖 Generated with Claude Code

Phase 3 (BugHunter). The full ablation eval (run_eval*.py — skills-on vs
skills-off on a self-grading target) needs Docker + Burp + an authed claude CLI,
so it runs locally, not in CI. But the harness around it can silently rot. This
adds the CI-able half plus a place to record the real numbers:

- .github/workflows/eval-harness.yml — on eval/ changes: py_compile every
  eval/*.py, run the oracle parser self-test, run the new harness sanity test.
  Network-free; contents: read.
- eval/test_harness.py — asserts (1) every eval/*.json config parses with the
  keys the runners expect, (2) oracle_portswigger classifies the lab-status
  widget correctly, (3) fp_app.py starts and serves every path in fp_cases.json
  (keeps the FP-trap app and its ground-truth cases in sync). Stdlib, localhost.
- eval/BASELINE.md — the local run procedure + a results table to fill in
  (skills-on/off solve-rate delta) and commit after a real run.

Building test_harness caught two bugs in my own probe (not the app): appending a
canary to an exact-match route corrupted it, and urllib auto-followed the
open-redirect trap's 302 to a 404. Fixed (query-aware URL + no-follow opener);
fp_app and fp_cases are in sync.

The real skills-on/off baseline is produced by running the eval locally and
committing BASELINE.md — CI can't run it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@elementalsouls
elementalsouls merged commit 4b10175 into main Aug 24, 2026
2 checks passed
@elementalsouls
elementalsouls deleted the eval/harness-sanity branch August 24, 2026 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant