Repository navigation
eval: fill v0 baseline with real skills-on/off ablation numbers - #75
Merged
Merged
Conversation
Ran the existing harness end-to-end (colima + Juice Shop + claude engine; no Burp needed for the curl-solvable challenge set). Model claude-opus-4-8, 3 challenges, both conditions: skills-off : 3/3 solved, median 2 turns, median $0.178 skills-on : 3/3 solved, median 3 turns, median $0.254 delta : +0 solve-rate (ceiling effect) Honest reading: Juice Shop is memorized + curl-trivial, so both conditions solve everything and the skill context is pure overhead — v0 validates pipeline/autonomy/cost, NOT skill value (exactly as the harness README warned). A real skill-delta needs the v1 PortSwigger-Academy tier (less memorized), which is gated on PS creds + manual lab launch. BASELINE.md was an empty template; now carries real numbers + the v1 path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ran the existing eval harness end-to-end (colima + Juice Shop + claude engine — no Burp needed for the curl-solvable set). Real v0: skills-off 3/3, skills-on 3/3 — ceiling effect (Juice Shop memorized), so v0 validates pipeline/autonomy/cost, not skill value (as the README warned). BASELINE.md was an empty template; now has real numbers + the v1 (PortSwigger, needs creds) path. No skill/code changes.