Repository navigation
fix(engine): keep recon redirects inside scope - #181
Open
fairoz-539 wants to merge 1 commit into
Open
fairoz-539 wants to merge 1 commit into
fairoz-539 wants to merge 1 commit into
Conversation
The engine's recon runs unattended, but its fetches followed redirects to any host: - spa_recon fetched each seed with redirects on, took the origin it landed on as `base`, and requested base + /robots.txt and /sitemap.xml with no scope check. An in-scope seed that 302s to an SSO provider, a deny-listed admin host or a third party got those hosts crawled. JS bundle fetches followed redirects the same way. - _probe ran `curl -L`, so platform_recon (.well-known, robots, Firebase), openapi_recon and prune_inert_params followed every redirect too; platform_recon and openapi_recon also probed seeds without checking them against scope. - auth_enforcement_sweep used urlopen, which followed a `302 -> /login` and graded the login page's 200 as "declared-secured, NOT enforced" -- a false positive, plus a request to wherever Location pointed. Fetch backends now make one hop at a time; _follow walks the chain (max 5 hops) only while the next URL passes the scope predicate and otherwise returns the last in-scope response. Callers pass scope.in_scope_host; with no predicate nothing is followed. Out-of-scope seeds are skipped by platform_recon and openapi_recon, and the auth sweep never follows redirects. This is the engine counterpart of the cbh fix in elementalsouls#149.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
The engine's recon phase runs unattended, and the module docstring says scope is enforced in code. Its HTTP fetches still followed redirects to any host:
spa_reconfetched each seed with redirects on, used the origin it landed on asbase, then requestedbase + /robots.txtandbase + /sitemap.xmlwith no scope check. An in-scope seed that 302s to an SSO provider, a deny-listed admin host or a third party got that host crawled. JS bundle fetches followed redirects the same way._proberancurl -L, soplatform_recon(.well-known, Firebase, robots),openapi_reconandprune_inert_paramsfollowed every redirect.platform_reconandopenapi_reconalso probed seeds without checking them against scope, unlikespa_recon.auth_enforcement_sweepusedurlopen, which follows redirects. A declared-secured op that answers302 → /logingot graded by the login page's200as "declared-secured, NOT enforced". That is a false positive (a redirect to login is enforcement), and it also sent a request whereverLocationpointed.This is the same class of bug #149 fixed in
cbh'srecon_http_probe, but in the engine.The fix
requests, stdlib,curl) now make one hop at a time._follow()walks the redirect chain, at mostMAX_REDIRECTS = 5hops, and only whileallow(next_url)accepts the next URL. Otherwise it returns the last in-scope response.scope.in_scope_host. With no predicate, nothing is followed (the safe default).platform_recon/openapi_reconskip out-of-scope seeds, the same wayspa_reconalready does./→/home, http → https on the same host) are still followed, so recon coverage on normal targets is unchanged.Type
Checklist
*.example.test/*.thirdparty.test/ loopback are used.scripts/scan_identifiers.py: 0 matches;scripts/check_doc_counts.py: 0 errors.engine/test_recon_redirect_scope.py(10 tests; 2 failures + 8 errors onmain, all pass here). They use a fake transport for exact scope assertions, plus real loopback-server tests for the stdlib backend, thecurlbackend (skipped if curl is missing) and the auth sweep. They pass with and withoutrequestsinstalled (CI has none). The fullengine/suite,tests/and every step ofruntime-tests.ymlpass. Code is Python 3.9-compatible.CHANGELOG.mdupdated.Anything reviewers should know?
engine/test_openapi_readonly.pypatchedurllib.request.urlopen, the exact call that followed redirects. It now patches the module's no-redirect opener (recon._OPENER.open). That is a one-line change, and the assertions are unchanged._probekeeps its(status, body)return shape, and_getkeeps(body, final_url, headers). Both only gained an optionalallow=argument.token_endpoint, etc.) are still recorded as surface items without a scope check. Nothing requests them, and the hunt phase already refuses out-of-scope items. Happy to filter them at recon time too if you prefer.CHANGELOG.md.