Skip to content

fix(report): preserve frontmatter in UTF-8 BOM Markdown files - #200

Closed
jazzpher wants to merge 2 commits into
elementalsouls:mainfrom
jazzpher:fix/report-utf8-bom
Closed

jazzpher wants to merge 2 commits into
elementalsouls:mainfrom
jazzpher:fix/report-utf8-bom

Conversation

@jazzpher

Copy link
Copy Markdown

What changed

Read finding Markdown with utf-8-sig so an optional leading BOM doesn't stop the parser from recognizing YAML frontmatter. This preserves values such as severity and asset in the generated draft. Plain UTF-8 Markdown and the output encoding are unchanged. No changes to recon manifest handling.

Tests

The regression test covers Markdown with and without a BOM. The BOM case fails on upstream 8258d29 and passes with the fix. The existing report section tests also pass.

PYTHONPATH=. python -m pytest -q tests/test_report_sections.py tests/test_report_utf8_bom.py

elementalsouls added a commit that referenced this pull request Nov 22, 2026
…CRLF, Windows console, h1 --program (#210)

* fix(cbh): triage gate — real request-target for Q1, scope-negation for Q3, clickjacking for Q7

Three false-verdict paths in the 7-Question Gate, each independently reported:

- Q1 matched a bare method word anywhere in the text, so prose like
  "target application" (contains "get ") or "input validation" passed Q1
  with no request at all. Q1 now needs a method followed by a path or URL
  (or curl / HTTP/1.1). Reported by @dajiaohuang (#209).
- Q7's never-submit list had "click-jacking" but not "clickjacking", so the
  unhyphenated spelling was never killed. Reported by @Ethereal49 (#197).
- Q3 answered YES on any scope/asset wording even when the finding itself
  said "out of scope" / "unverified". A scope/asset line carrying a negation
  no longer counts as in-scope evidence; non-scope lines ("Impact: unclear")
  still don't. Reported by @Ethereal49 (#197).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cbh): parse finding frontmatter through a leading UTF-8 BOM

A BOM (\ufeff) written by some Windows editors sits before the `---` fence,
so the `^---` frontmatter regex missed it and severity/asset fell back to
defaults. Strip a leading BOM in parse_finding_metadata, the shared chokepoint
for every command that reads a finding, not just `cbh report`.

Reported by @jazzpher (#200).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cbh): force UTF-8 stdout so Windows cp1252 consoles don't crash

cbh and setup_harness_mcp.py print the gate glyphs (check / cross / arrow /
em-dash) throughout. On a Windows console (cp1252) a bare print() raised
UnicodeEncodeError and the command died mid-run. Reconfigure stdout/stderr to
UTF-8 at both entrypoints, the same fix engine/engine.py:main already uses, so
the whole CLI is covered at one point rather than guarding every print.

Reported by @Sahilkumar-r (#180).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(offensive-osint): filter h1_reference --program by team_handle, not team_id

CompleteHacktivityReportIndex no longer accepts terms: {team_id: ...}, so
--program silently returned nothing. Switch to query_string "team_handle:<h>",
which filters correctly and still accepts sort + pagination (so --program now
composes with --top-voted / --top-bounty). Also: route the lookup progress line
to stderr so --json stdout stays pipeable, and reject an empty --program handle.

Two independent reports converged on this fix: @roonnnnyyyyyy (#206) and
@flipcyde101 (#202).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(changelog): record the four tooling bugfixes with reporter credits

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cbh,osint): review follow-ups — Q3 negation, Q1 markdown, CRLF, canonical handle

Adversarial review of the branch surfaced five real defects:

- Q3 _q3_scope_negated matched ANY line containing 'scope' plus a negation,
  so 'Impact: unclear scope of data exposure' wrongly killed an in-scope
  finding. Negation is now evaluated per line: strong phrases ('out of scope')
  disavow anywhere; weak words ('unknown','unclear') only on a scope/asset/
  target line (the key before the colon).
- Q3 negation list was space-only, so the common hyphenated 'out-of-scope'
  slipped through. Hyphens are normalized to spaces before matching.
- Q1 required whitespace between method and path, missing the markdown a PoC
  is usually written in (`GET /x`, **GET** /x, GET `/x`). The separator class
  now allows backticks and bold/italic markers. The prose ceiling
  ('an attacker can get /admin') is marked, not closed — requiring HTTP/ would
  reject the bare 'GET /path' PoC the gate must accept.
- parse_finding_metadata now also normalizes CRLF, so Windows findings (the
  same platform as the BOM in #200) don't skip frontmatter.
- h1_reference --program queried the raw input handle; query_string is
  case-sensitive while TeamLookup is not, so '--program Shopify' could return
  nothing. lookup_program now returns the team record and main uses the
  canonical handle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@elementalsouls

Copy link
Copy Markdown
Owner

Thanks @jazzpher — the BOM-before---- bug is fixed in 9ba66b1 (#210). Took it at the shared parse_finding_metadata chokepoint (covers every command, not just cbh report) and also normalized CRLF while there, same Windows platform. Credited in CHANGELOG. Closing as built-in-house.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants