Broaden and harden PoC payload recovery; generate every quoted figure from the corpus - #30
Open
AntoineBastide47 wants to merge 27 commits into
Open
Broaden and harden PoC payload recovery; generate every quoted figure from the corpus#30AntoineBastide47 wants to merge 27 commits into
AntoineBastide47 wants to merge 27 commits into
Conversation
Fetch seven new sources, pinned by commit in manifest.json:
trickest/cve, vulhub CVE -> PoC linkage
ARPSyndicate/cve-scores VEDAS exploit-availability scores
fuzzdb, PayloadsAllTheThings attack payloads by sink type
SecLists "
OWASP BenchmarkJava labeled injection cases
plus the NIST SARD Juliet Java and C/C++ suites, whose paths encode the
CWE. New tables: payload, ground_truth, exploit_score. A new validate
stage runs VERIFY over the payload corpora, which measure matchertext
directly instead of only through CVEs that happen to expose a payload,
and resolves the labeled CWEs through the taxonomy anchors.
fetch.py gains --freeze so an already-pinned rolling feed stays at its
recorded revision, making a corpus change attributable to new sources
rather than to upstream drift.
Two details that would otherwise distort counts:
- a trickest entry separates candidate exploit repos from advisory
write-ups; only the former count as PoC evidence, worth 54% of the
rows.
- vulhub's local_path points at the environment README, which carries
the worked exploit request, not at the directory.
fix: make payload selection deterministic
subclass, syntactic_group, and report picked a CVE's representative
payload from an unordered query, so ingesting a new PoC source silently
reshuffled which file each CVE extracted from. Seeding report's RNG did
nothing while the list it shuffled had no fixed order. All three now
order by (cve_id, source, ref); two consecutive runs are byte-identical.
feat: extract URL-form payloads
The literal signatures required real whitespace, so union+select,
UNION/**/SELECT, and %20UNION%20SELECT all missed; URL forms were the
largest extraction gap. A URL_SIGNS layer, tried only after the literal
ones, matches those forms and decodes the match: percent-escapes (two
rounds, since %2527 -> %27 -> ' is common), + as query-string space, and
/**/ as a whitespace substitute. Decoding is required for correctness,
not convenience -- the server decodes before the sink parses, so %28 is
really a ( and skeletonizing the raw form would hide matchers.
Also adds attribute- and JS-context XSS breakout, which carry no tag for
the old signatures to anchor on, and URL forms for shell_command and
code_eval. Those last are needed for the measurement to be sound:
improving only SQL and XSS, both matcher-hostable, skewed the
payload-backed set and inflated the contained share to 87%.
Extraction rises 42% -> 63%, payload-backed CVEs 5,040 -> 7,474. The
inert share of contained falls 91.8% -> 91.0% as decoding exposes
unmatched parens, the direction that confirms the change does not favour
a flattering result.
pipeline/latex.py writes doc/generated/, which main.tex inputs: 104
scalar macros plus the syntax, mechanism, payload-corpus, PoC-source and
ground-truth tables, and the containment tree. The tree is emitted as a
whole verbatim block because verbatim does not expand macros.
No number is hand-copied any more; a scan for {,}-formatted literals
outside generated/ comes back empty. Regenerate with
python3 pipeline/latex.py, then make.
Validated by reproducing the previous hand-computed values exactly where
the data had not changed: the tab:mech hostable block, its totals, the
whole projection chain (73,246 -> 7.3% -> 69,536 -> 76%), 351 distinct
CWEs, 345 KEV, and the naive-Bayes holdout scores.
That surfaced six figures the corpus rebuild had already invalidated,
now corrected: injection CVEs with a PoC 13,691 -> 52,684, payload-backed
5,025 -> 7,474, outside 914 -> 1,086, skeleton groups 2,143 -> 3,906,
singletons 1,817 -> 3,347, and the whole "with PoC" column of tab:syntax.
data.tex said four PoC databases; there are now six.
A signature that stops inside a function call, as in `ASCII(SUBSTRING(passwd,`, leaves an unmatched opener that VERIFY then reports as a rejection. 212 of 573 rejected CVEs (37%) were this artifact rather than a real breakout, so the published inert/reject split was wrong. The two cases separate cleanly on which side is unbalanced. A breakout must emit an unmatched *closer* -- closing the slot early is the whole manoeuvre -- so `'r0t')</script>` is genuine. An unmatched *opener* at the tail is always the capture being cut short. Extraction now discards the latter. Checked against the corpus: all 341 excess-closer rejections are real payloads, all 212 excess-opener ones were truncated. Also drops three false-positive classes that markdown prose produces and exploit files did not, found via the GitHub pilot: inline code spans (`deviceList` read as command substitution), `&id=62` read as the id command, and bare `javascript:` stubs. And adds the paren-breakout form `admin') OR '1'='1`, which the quote-anchored signature missed -- these are exactly the unmatched-closer cases the split turns on. payload-backed 7,474 -> 7,221 rejected 573 -> 347 reject share 9.0% -> 5.6% junk in corpus 4% -> ~0% feat: add GitHub PoC fetch pilot Measures reach, yield and syntax skew on a sample before committing to a bulk fetch. Its first result corrects the addressable set: most GitHub links point at CVE mirrors, not exploits (one nuclei-templates fork is linked from 11,758 CVEs), so repos above a fan-out threshold are dropped and each CVE is represented by its most specific repo. That leaves ~12,600 addressable CVEs, not the 30,608 a raw link count suggests.
A linked PoC repo is often the vulnerable application itself, not a proof
of concept, and scanning its source recovered benign code as payloads:
jQuery includes, Vite entry points, Blade templates, a C4 architecture
diagram whose System(...) call matched the code-eval signature, and CLI
flags like -lzma2=dict=64KiB matching argument injection. Half the
payloads from a 40-repo trial were of this kind.
Curated exploit files hid the problem because everything in them is an
exploit. Arbitrary repository source is not, so the loose signatures had
to become discriminating:
<script> require a body that alerts, evals or touches
document.cookie, not any script tag
"><script require the same, not an attribute boundary
javascript: require an attack call, not an inline handler
system()/eval() require a shell command, $, backtick or
base64_decode as the argument, not an identifier
-flag=value require a command substitution or separator
fetch_github only searches documentation, plus code whose path claims to
be the exploit. That costs some genuine payloads and removes far more
application source.
Also reclaims closers a regex cannot count: eval(base64_decode($x)) was
captured one ) short and then discarded by the truncation guard, so the
capture is now extended over closing matchers that immediately follow.
Junk in sampled payloads falls from ~50% to ~8%; yield drops to 22%,
which is the intended trade.
feat: pin fetched repos by commit SHA
git ls-remote resolves HEAD without spending API quota, and codeload
serves that exact commit, so the archive root is repo-<sha> and the pin
proves itself. Pins land in data/github_pins.json, making the fetch
replayable even though a quarter of these repos are already gone.
Payloads land in remote_payload and feed syntactic_group and subclass,
which prefer a local file and fall back to the remote payload.
Downloading archives was bandwidth-bound, not concurrency-bound. Twenty four parallel codeload fetches saturated the link at ~283 KB/s, so neither more threads nor more cores moved it; measured throughput was 1.66/s however the work was arranged. Two earlier speedups I reported were measurement artifacts: a startup burst read as steady state, and a poll that straddled exactly one 200-row flush, which is quantised because flush only writes every 200 results. One GraphQL request now returns the SHA, the root file listing and the README for a batch of repos; a second fetches only the documents worth reading. A few KB per repo replaces a multi-megabyte archive, which takes the transfer off the critical path entirely. throughput 1.66/s -> 38/s (12,590 repos in 2m50s, was ~2h) yield 28% -> 31% of reachable repos oversize 348 -> 0, since nothing large is transferred Reading ten root documents per repo beats the archive scan outright: the archive searched more files but the payload is nearly always in the write-up, and skipping application source avoids its false positives. Corpus effect of the completed fetch: payload-backed CVEs 7,221 -> 9,633 contained 6,244 -> 7,937 contained share 86.5% -> 82.4% rejected 347 -> 600 junk in payloads 0.0% The contained share falls because the recovered payloads are not the same mix as the local corpus: non-matcher contexts triple (334 -> 1,021), mostly shell commands documented in PoC write-ups. That corrects a skew in the previous figure rather than contradicting it, and lands close to the 82% originally reported before any of this work. 5,595 commit SHAs pinned in data/github_pins.json, so the fetch replays even though 3,034 of the linked repos (24%) are already deleted.
An audit of every numeric literal across the paper found six data
figures the generator did not cover, one of them already stale:
concl.tex 82% contained, 80% hostable -- both headline results,
and 82% was only correct by coincidence; it read 86.5%
two commits ago
results.tex "fifteen distinct verdicts" -- now 16, changed by the
GitHub fetch
results.tex "four in five" for the two dominant syntaxes, now
emitted as a share
classify.tex code/eval naive-Bayes scores, the deterministic layers'
share of labels, and the audit sample size
The audit size is a pipeline parameter rather than a measurement, so
export.AUDIT_PER_METHOD is now the single definition and latex.py reads
it from there.
A scan for data-shaped literals outside generated/ is now empty across
abs, corpus, results, prevent, concl, classify and data. corpus.tex
quotes no figures at all.
Payload-backed CVEs 9,633 -> 11,087, junk 0.08%, contained share
unchanged at 82.4%.
1. Exploit-DB links joined to the local clone (+3,086 poc rows)
load_exploitdb mapped CVEs through the `codes` column of
files_exploits.csv; where that column is blank the mapping was lost even
though another source linked the CVE to the same EDB entry and the file
was already on disk. Pure bookkeeping, no network.
2. Descend into doc/poc subdirectories (+148)
Well below the 10-15% I estimated: only 8 of 60 previously-empty repos
have such a directory at all, so the yield is ~2%. The first DIR_HINT
also matched by substring, taking `docker` for a doc directory and
`aj-report` for a report; it is anchored on name segments now.
3. Widen the aggregator cut-off to fan-out 100 (+337)
13% yield against 31% for the fan-out<=20 band, so this range is a mix
of genuine multi-CVE researcher repos and mirrors. Still worthwhile.
4. Trust explicit payload labels (+~870)
Exploit-DB write-ups and sqlmap transcripts label the payload outright.
The label supplies what a signature cannot: it marks a string as the
attack even when the sink is one the signatures do not enumerate, as in
`...¤tTSREmailTo=|date>/tmp/x`. Applied only after every signature
has failed, and the value must still carry a delimiter or metacharacter,
contain an `=`, and pass the junk guards.
Two ordering details matter for correctness. The whole labelled value is
preferred over a signature match inside it, because the signature
captures a suffix such as `SLEEP(5)))vltp)` whose stray closers turn a
balanced payload into an apparent rejection. And is_truncated is not
applied to labelled values: it detects a regex stopping mid-expression,
whereas a labelled line is a delimited whole, so the trailing `{` of
`'};alert(1);{'` is the payload rather than a cut.
The SLEEP signature no longer runs to end of line for the same reason.
That correction alone drops the reject share from 7.6% to 5.6%.
…orm block fetch_web reads the two PoC sources that host advisory text rather than another link to it, and that need no API key. seclists completed: 374 pages, 178 payloads (48%), 165 of them CVEs with no other source. Payload-backed CVEs 11,087 -> 11,265. Packet Storm blocked us partway through and its rows have been deleted. Two defects caused it, both mine: The throttle and the lane partition keyed on hostname. Packet Storm answers to both packetstormsecurity.com and .org, so one server received two independent one-per-second budgets and thus 2/s for twenty-five minutes, which tripped its velocity limiter. Both now key on the site. The block is served as HTTP 200 with a refusal page, so every request after it was recorded as "this advisory contains no payload". That is worse than an error: 1,093 CVEs carried a false negative that looked like a measurement. Responses are now checked for refusal text, and a site is abandoned after three consecutive blocks instead of being hammered for the rest of the run. Packet Storm is not retried here. We are blocked, the polite interval is unknown, and re-running against a site that has explicitly refused us is not something to do casually; its ~1,500 CVEs stay unaddressed.
AntoineBastide47
force-pushed
the
CVE-better-PoC-usage
branch
from
July 28, 2026 17:14
0bd26bf to
964bb19
Compare
prevent.tex asserted that the distance between the payload-backed subset and the full corpus is payload availability rather than a containment failure, but gave the reader nothing to size it with. Two published measurements do: Mandiant puts 51% of vulnerabilities disclosed in 2021-2022 as eventually acquiring public exploit or proof-of-concept code, and VulnCheck puts 26% of CVEs with 2025 identifiers as having it within the year, a share that grows with age. Our own corpus agrees from the other side: 57% of injection CVEs carry any proof-of-concept reference at all, so for the rest there is no artifact for any pipeline to recover. A literal attack string is stricter again, since a proof of concept that builds its payload at run time exposes none. The point of the paragraph is that the fragment-level bound should be read against the measurable population, roughly half the corpus, rather than against all 91,760. These are literature constants rather than corpus measurements, so they stay as literals with citations instead of becoming generated macros; the two corpus figures in the same paragraph are macros as usual.
AntoineBastide47
force-pushed
the
CVE-better-PoC-usage
branch
from
July 29, 2026 08:20
964bb19 to
e6a11ca
Compare
The closing of the projection called the reach "bounded from below at the fragment level and from above at the syntax level" and then gave 82% and at most 80%. Those measure different populations: 82% is contained over the payload-backed subset, 80% is matcher-delimitable syntax over the whole corpus. Presented as a range the floor exceeds the ceiling, which reads as a contradiction unless the reader notices the denominators differ. Adds \dataFloorShare, contained over all injection CVEs, so the three figures sit on one scale: a floor of 10.1%, a central estimate of 77%, a ceiling of 80%. The floor treats every CVE whose payload we could not recover as uncontained, making it a hard lower bound rather than an estimate; the paragraph that follows explains why that assumption is harsh, since most of those CVEs have no published payload at all. ContainedShare stays, relabelled as what it is: what happens once a payload is known, not how much of the corpus that covers. The abstract and conclusion already qualified both denominators in prose and are unchanged.
…yntax The projection took the count of matcher-delimitable CVEs and subtracted an estimate of javascript: sinks, a correction hand-written for HTML/DOM and applied nowhere else. That happened to be right, but only because HTML/DOM is the single syntax where the model can find a partial rate; it does not generalise, and it obscured why the correction is needed. The reason is sampling. Payload-backed CVEs are not a random draw: SQL is 1.7x over-represented because a SQL payload is a literal string a write-up can quote, while code-eval and argument payloads are built at run time and never appear verbatim. Payload availability correlates with syntax, and specifically with matcher-hostable syntax, 85% of the subset against 80% of the corpus. Applying the pooled 82% to the whole corpus therefore yields 75,550, too high by 5,276 CVEs. latex.post_stratify now measures a containment rate per syntax and weights by the corpus counts. The estimate is unchanged at 70,274 (77%), which is the point: the two routes agree exactly, so the syntax-level bound is the post-stratified form of the fragment-level measurement rather than an independent guess. A syntax with no payload-backed CVE falls back to the structural rule. Adds tab:strata showing the strata, and \dataNaiveEstimate and \dataSkewCost so the text can state what the skew costs rather than asserting the correction matters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Payload-backed injection CVEs 5,025 → 11,265, so an increase by a factor of x2.2.
Sources. Nine added to
fetch.py, pinned inmanifest.json: trickest/cve, vulhub, cve-scores, fuzzdb/PayloadsAllTheThings/SecLists (25,369 payloads by sink type), OWASP Benchmark + NIST SARD Juliet (149,811 CWE-labeled cases). Newfetch_github.py(15,194 repos, 5,692 SHAs pinned) andfetch_web.py(seclists, content-hashed). Newvalidate.pyruns VERIFY over the payload corpora directly and checks the taxonomy against the labeled suites: all 15 injection CWEs resolve as expected.Several measurement bugs, some of which corrupted numbers on the base branch:
union+select,%20UNION%20SELECT) were invisible.Junk 4% → 0.08%. Reject share 9.0% → 5.6%, mostly artifacts.
Generated figures.
latex.pyemitsdoc/generated/: 110 macros, five tables, the containment tree. No data literals remain outsidegenerated/. Validated by reproducing the base branch's values exactly where data was unchanged, which surfaced six already-stale figures, including both numbers in the conclusion.P.S: requires #29 to be merged first