Skip to content

fix(engine): keep recon redirects inside scope - #181

Open
fairoz-539 wants to merge 1 commit into
elementalsouls:mainfrom
fairoz-539:fix/recon-redirects-stay-in-scope
Open

fairoz-539 wants to merge 1 commit into
elementalsouls:mainfrom
fairoz-539:fix/recon-redirects-stay-in-scope

Conversation

@fairoz-539

Copy link
Copy Markdown

What does this change?

The engine's recon phase runs unattended, and the module docstring says scope is enforced in code. Its HTTP fetches still followed redirects to any host:

  1. spa_recon fetched each seed with redirects on, used the origin it landed on as base, then requested base + /robots.txt and base + /sitemap.xml with no scope check. An in-scope seed that 302s to an SSO provider, a deny-listed admin host or a third party got that host crawled. JS bundle fetches followed redirects the same way.
  2. _probe ran curl -L, so platform_recon (.well-known, Firebase, robots), openapi_recon and prune_inert_params followed every redirect. platform_recon and openapi_recon also probed seeds without checking them against scope, unlike spa_recon.
  3. auth_enforcement_sweep used urlopen, which follows redirects. A declared-secured op that answers 302 → /login got graded by the login page's 200 as "declared-secured, NOT enforced". That is a false positive (a redirect to login is enforcement), and it also sent a request wherever Location pointed.

This is the same class of bug #149 fixed in cbh's recon_http_probe, but in the engine.

The fix

  • The fetch backends (requests, stdlib, curl) now make one hop at a time. _follow() walks the redirect chain, at most MAX_REDIRECTS = 5 hops, and only while allow(next_url) accepts the next URL. Otherwise it returns the last in-scope response.
  • Every recon caller passes scope.in_scope_host. With no predicate, nothing is followed (the safe default).
  • platform_recon / openapi_recon skip out-of-scope seeds, the same way spa_recon already does.
  • The auth sweep never follows redirects, so a 3xx is graded as itself.
  • In-scope redirects (/ → /home, http → https on the same host) are still followed, so recon coverage on normal targets is unchanged.

Type

  • New skill
  • Improvement to an existing skill
  • Docs / examples
  • Tooling / scripts / CI
  • Other:

Checklist

  • No client/engagement data. Only *.example.test / *.thirdparty.test / loopback are used.
  • scripts/scan_identifiers.py: 0 matches; scripts/check_doc_counts.py: 0 errors.
  • Tests: new engine/test_recon_redirect_scope.py (10 tests; 2 failures + 8 errors on main, all pass here). They use a fake transport for exact scope assertions, plus real loopback-server tests for the stdlib backend, the curl backend (skipped if curl is missing) and the auth sweep. They pass with and without requests installed (CI has none). The full engine/ suite, tests/ and every step of runtime-tests.yml pass. Code is Python 3.9-compatible.
  • CHANGELOG.md updated.

Anything reviewers should know?

  • engine/test_openapi_readonly.py patched urllib.request.urlopen, the exact call that followed redirects. It now patches the module's no-redirect opener (recon._OPENER.open). That is a one-line change, and the assertions are unchanged.
  • _probe keeps its (status, body) return shape, and _get keeps (body, final_url, headers). Both only gained an optional allow= argument.
  • Not changed: OAuth discovery endpoints (token_endpoint, etc.) are still recorded as surface items without a scope check. Nothing requests them, and the hunt phase already refuses out-of-scope items. Happy to filter them at recon time too if you prefer.
  • This PR is independent of the scope-rule normalization PR. The two only touch the same lines in CHANGELOG.md.

The engine's recon runs unattended, but its fetches followed redirects
to any host:

- spa_recon fetched each seed with redirects on, took the origin it
  landed on as `base`, and requested base + /robots.txt and
  /sitemap.xml with no scope check. An in-scope seed that 302s to an
  SSO provider, a deny-listed admin host or a third party got those
  hosts crawled. JS bundle fetches followed redirects the same way.
- _probe ran `curl -L`, so platform_recon (.well-known, robots,
  Firebase), openapi_recon and prune_inert_params followed every
  redirect too; platform_recon and openapi_recon also probed seeds
  without checking them against scope.
- auth_enforcement_sweep used urlopen, which followed a
  `302 -> /login` and graded the login page's 200 as
  "declared-secured, NOT enforced" -- a false positive, plus a
  request to wherever Location pointed.

Fetch backends now make one hop at a time; _follow walks the chain
(max 5 hops) only while the next URL passes the scope predicate and
otherwise returns the last in-scope response. Callers pass
scope.in_scope_host; with no predicate nothing is followed.
Out-of-scope seeds are skipped by platform_recon and openapi_recon,
and the auth sweep never follows redirects.

This is the engine counterpart of the cbh fix in elementalsouls#149.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant