Skip to content

eval: fill v0 baseline with real skills-on/off ablation numbers - #75

Merged
elementalsouls merged 1 commit into
mainfrom
eval-v0-baseline
Aug 24, 2026
Merged

elementalsouls merged 1 commit into
mainfrom
eval-v0-baseline

Conversation

@elementalsouls

Copy link
Copy Markdown
Owner

Ran the existing eval harness end-to-end (colima + Juice Shop + claude engine — no Burp needed for the curl-solvable set). Real v0: skills-off 3/3, skills-on 3/3 — ceiling effect (Juice Shop memorized), so v0 validates pipeline/autonomy/cost, not skill value (as the README warned). BASELINE.md was an empty template; now has real numbers + the v1 (PortSwigger, needs creds) path. No skill/code changes.

Ran the existing harness end-to-end (colima + Juice Shop + claude engine; no Burp
needed for the curl-solvable challenge set). Model claude-opus-4-8, 3 challenges,
both conditions:

  skills-off : 3/3 solved, median 2 turns, median $0.178
  skills-on  : 3/3 solved, median 3 turns, median $0.254
  delta      : +0 solve-rate (ceiling effect)

Honest reading: Juice Shop is memorized + curl-trivial, so both conditions solve
everything and the skill context is pure overhead — v0 validates pipeline/autonomy/cost,
NOT skill value (exactly as the harness README warned). A real skill-delta needs the
v1 PortSwigger-Academy tier (less memorized), which is gated on PS creds + manual lab
launch. BASELINE.md was an empty template; now carries real numbers + the v1 path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@elementalsouls
elementalsouls merged commit 02ece8f into main Aug 24, 2026
2 checks passed
@elementalsouls
elementalsouls deleted the eval-v0-baseline branch August 24, 2026 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant