Repository navigation
fix(scope): stop out-of-scope rules from silently failing open - #182
fairoz-539 wants to merge 2 commits into
Conversation
A deny rule only works if it can match the host it names. Several common ways of writing an out-of-scope asset matched nothing, so the host fell through to a broader in-scope rule and the engine tested it: out_of_scope: admin.example.com:8443 (port) out_of_scope: https://admin.example.com (scheme, scope.json / CLI) out_of_scope: admin.example.com/panel (path) out_of_scope: admin.example.com. (trailing dot) out_of_scope: 2001:db8::/32 (IPv6 CIDR) out_of_scope: re:^admin[.example\.com$ (invalid regex -> skipped) Normalize every rule once, in Scope, whatever its source (scope.md, scope.json, --in-scope/--out-of-scope, osint/recon helpers): reduce it to its host, canonicalize IPv4/IPv6 addresses and CIDRs, and record a warning when a path or port is dropped or a rule still cannot match. An out-of-scope regex that does not compile now raises ValueError (exit 2 from scope.py) instead of being skipped. Warnings are printed by scope.py and written to the engine log. scope.json is read as UTF-8. The scope.md-only URL reduction from elementalsouls#155 is now the general rule, so _rule_from_token is removed.
auditbenchaibot
left a comment
There was a problem hiding this comment.
🤖 Automated review by Audit Bench Ai, not a personal review. It was posted through the GitHub account connected to Audit Bench Ai. Install the Audit Bench Ai GitHub App to have reviews posted by a separate bot account.
No actionable comments were generated in this review. 🎉
Merge risk: 🟢 Low · up to 483ed60
Targets previously matched by lenient in-scope fallthrough may now be blocked or cause errors if invalid deny regexes are provided, but this strictly enforces intended safety boundaries.
Merge readiness: 🟡 Ready to merge, with notes
Pre-merge checks: ✅ 8 · ⚠️ 2
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| CI checks | No CI checks ran on this commit. | Add CI so every PR is built and tested. | |
| Tests | 3 source files changed but no test files did. 📄 engine/scope.py📄 engine/engine.py📄 engine/test_scope_rule_normalization.py |
Add or update tests that cover the new behaviour. |
✅ Passed checks: 8 passed
| Check name | Status | Explanation |
|---|---|---|
| Merge conflicts | ✅ | Merges cleanly into main. |
| Secrets | ✅ | No keys, passwords or tokens in the changed code. |
| Blocking issues | ✅ | No critical or high-severity issues in the changed lines. |
| PR size | ✅ | 305 lines changed. |
| Title check | ✅ | The title clearly states that out-of-scope rules are prevented from failing open. |
| Description check | ✅ | The description provides detailed context, reproduction tables, and an explanation of the fix. |
| Linked issues check | ✅ | Mentions related PRs #155 and #160 as background context, and the implemented changes directly follow that lineage without unresolved issue links. |
| Out of scope changes check | ✅ | All modifications in the engine, scope module, and test suite directly relate to scope rule normalization and enforcement. |
📝 Walkthrough
This pull request fixes an issue where out-of-scope rules with ports, paths, schemes, or formatting quirks failed to match and silently allowed targets to remain in-scope. It introduces centralized rule normalization in Scope.init, adds IPv6 CIDR and canonical address matching, and raises a ValueError on uncompilable out-of-scope regex rules to fail closed. In addition, rule widening or invalid patterns now generate warnings logged by the engine and CLI.
| File | Changes |
|---|---|
engine/engine.py |
Logs warnings emitted during scope normalization on engine startup. |
engine/scope.py |
Implements centralized rule normalization, canonical IP/IPv6 CIDR handling, fail-closed deny regex parsing, and warning collection. |
engine/test_scope_rule_normalization.py |
Adds comprehensive tests verifying deny rule forms, IPv6 matching, regex error handling, and scope warnings. |
ℹ️ Review info
- Commit:
483ed60 - Verdict: ✅ Pass
- Findings: none
- Review: local checks plus AI review of the changed lines
- Scope: Reviewed 3/3 selected files (of 3 total): 3 reviewed by AI, 0 from cache, 0 passed local checks with nothing risky enough to send to AI. Found 0 finding(s), No secrets found.
💬 Reply to any inline comment and Audit Bench Ai will recheck that code and answer in the thread. Mention @auditbench in the conversation to ask about the whole PR.
🛠️ Open the full report and fix with AI
Thanks for using Audit Bench Ai! Free pull request reviews for open source: paste any public PR at auditbenchai.com/scan.
Reviewed by Audit Bench Ai · Build fast. Check what ships. · Review any public PR free
Resolve the engine/scope.py conflict with elementalsouls#191: both sides read scope.json as UTF-8; keep the with-block form so the file is closed. Drop the now-redundant UTF-8 note from the CHANGELOG entry.
auditbenchaibot
left a comment
There was a problem hiding this comment.
🤖 Automated review by Audit Bench Ai, not a personal review. It was posted through the GitHub account connected to Audit Bench Ai. Install the Audit Bench Ai GitHub App to have reviews posted by a separate bot account.
Actionable comments posted: 1
Merge risk: 🟡 Medium · up to 3595415
URL-formatted IP rules with subnets may incorrectly strip netmasks, and regex matching failures at runtime could bypass deny protections unexpectedly.
Merge readiness: ⛔ Not ready to merge
Pre-merge checks: ✅ 7 · ⚠️ 2 · ❌ 1
❌ Failed checks: 1 error, 2 warnings
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| CI checks | No CI checks ran on this commit. | Add CI so every PR is built and tested. | |
| Blocking issues | ❌ | 1 critical issue in the changed lines. 📄 engine/scope.py:99 |
Fix the critical inline comments before merging. |
| Tests | 3 source files changed but no test files did. 📄 engine/scope.py📄 engine/engine.py📄 engine/test_scope_rule_normalization.py |
Add or update tests that cover the new behaviour. |
✅ Passed checks: 7 passed
| Check name | Status | Explanation |
|---|---|---|
| Merge conflicts | ✅ | Merges cleanly into main. |
| Secrets | ✅ | No keys, passwords or tokens in the changed code. |
| PR size | ✅ | 305 lines changed. |
| Title check | ✅ | The title clearly states that out-of-scope rules are being fixed to stop failing open. |
| Description check | ✅ | The description provides comprehensive background, reproduction examples, and details of the changes. |
| Linked issues check | ✅ | References #155 and #160 for background context on past normalization and encoding fixes, which align with this implementation. |
| Out of scope changes check | ✅ | All changes directly pertain to scope parsing, normalization, warning logs, and relevant unit tests. |
📝 Walkthrough
This pull request ensures scope rules (especially out-of-scope deny rules) are consistently normalized regardless of input format or source. Rules specifying schemes, ports, paths, or IPv6 formats are mapped to canonical host or CIDR patterns rather than silently failing open. Additionally, invalid deny regex patterns now fail closed by raising an error instead of being skipped.
| File | Changes |
|---|---|
engine/engine.py |
Logs scope normalization warnings when the engine starts. |
engine/scope.py |
Adds universal rule normalization, IPv6 CIDR support, and fails closed on invalid deny regexes. |
engine/test_scope_rule_normalization.py |
Adds comprehensive unit tests verifying scope rule normalization, regex failure modes, and warning behavior. |
ℹ️ Review info
- Commit:
3595415 - Verdict:
⚠️ Needs work - Findings: 1 critical, 1 high
- Review: local checks plus AI review of the changed lines
- Scope: Reviewed 3/3 selected files (of 3 total): 2 reviewed by AI, 1 from cache, 0 passed local checks with nothing risky enough to send to AI. Found 2 finding(s), No secrets found.
🤖 Prompt to fix review comments
Verify each finding against the current code and only fix it if it still applies.
In `@engine/scope.py` around lines 99 - 103: Invalid or uncompilable out-of-scope regex fails open instead of blocking targets. When matching a pattern in `_match`, if a regex pattern (`re:...`) fails to compile or throws an error during `re.search`, the exception is swallowed and returns `False`. If this occurs during the evaluation of an out-of-scope rule in `in_scope_host` (`any(_match(p, host) for p in self.out_of_scope)`), returning `False` means the deny rule does not match, allowing the target to be considered in-scope. This violates the core design principle that deny rules must never fail open. Suggested fix: Let `re.error` propagate or treat invalid regex evaluation in deny rules as matching (or failing closed) so that scanning agents are not inadvertently permitted to access targets when an out-of-scope regex fails.
In `@engine/scope.py` around lines 74 - 77: Subnet masks in URL-formatted rules are stripped into single host IPs. When an IP CIDR block is supplied in URL format (such as `http://10.0.0.0/24` or `10.0.0.0/24`), `_canonical_ip(t)` on line 71 handles bare CIDRs like `10.0.0.0/24`. However, if the rule has a scheme like `http://10.0.0.0/24` or extra URL components, it bypasses `_canonical_ip(t)` and is parsed via `urlparse('//' + t)`. In that case, `urlparse` treats `/24` as `parsed.path` and `parsed.hostname` becomes just `10.0.0.0`. The rule is normalized to the single IP `10.0.0.0`, discarding the `/24` subnet entirely and leaving the rest of the subnet unprotected by out-of-scope deny rules. Suggested fix: Before parsing with `urlparse`, check if the host component contains an IP network CIDR, or re-attach the path if the host is an IP address and `parsed.path` starts with a valid network prefix length.
💬 Reply to any inline comment and Audit Bench Ai will recheck that code and answer in the thread. Mention @auditbench in the conversation to ask about the whole PR.
🛠️ Open the full report and fix with AI
Thanks for using Audit Bench Ai! Free pull request reviews for open source: paste any public PR at auditbenchai.com/scan.
Reviewed by Audit Bench Ai · Build fast. Check what ships. · Review any public PR free
| try: | ||
| parsed = urlparse(t if "://" in t or t.startswith("//") else "//" + t) | ||
| host = (parsed.hostname or "").rstrip(".") | ||
| port = parsed.port |
There was a problem hiding this comment.
🤖 Audit Bench Ai (automated) | 🧠 Logic | 🟠 High
Subnet masks in URL-formatted rules are stripped into single host IPs
When an IP CIDR block is supplied in URL format (such as http://10.0.0.0/24 or 10.0.0.0/24), _canonical_ip(t) on line 71 handles bare CIDRs like 10.0.0.0/24. However, if the rule has a scheme like http://10.0.0.0/24 or extra URL components, it bypasses _canonical_ip(t) and is parsed via urlparse('//' + t). In that case, urlparse treats /24 as parsed.path and parsed.hostname becomes just 10.0.0.0. The rule is normalized to the single IP 10.0.0.0, discarding the /24 subnet entirely and leaving the rest of the subnet unprotected by out-of-scope deny rules.
Why it happens: urlparse splits the CIDR prefix into the path component when parsing URLs containing CIDRs, and _normalize_rule assigns host = parsed.hostname.
When it breaks: An out-of-scope rule is entered as https://192.168.1.0/24. Normalization reduces the rule to 192.168.1.0. A target at 192.168.1.5 is checked; it does not match 192.168.1.0, bypassing the out-of-scope restriction entirely.
Tests: Test test_regex_and_cidr_rules_are_untouched only tests bare CIDRs like 2001:db8::/32, not CIDRs entered with a scheme or URL prefix.
Suggested fix: Before parsing with urlparse, check if the host component contains an IP network CIDR, or re-attach the path if the host is an IP address and parsed.path starts with a valid network prefix length.
95% confidence · 💬 Reply here and Audit Bench Ai will recheck the code · 👀 to acknowledge · 👍/👎 teaches it what your team wants
What does this change?
engine/scope.pyis the deterministic gate the engine and/scoperely on ("deny wins, default deny"). Several common ways of writing an out-of-scope asset match no host. The deny rule is skipped without any error, so the host falls through to a broader in-scope rule and gets tested.Repro on
main(Scope(in_scope=["example.com"], out_of_scope=[RULE]), targethttps://admin.example.com/):mainadmin.example.com:8443(port, common on program pages)https://admin.example.com(scheme,scope.json/--out-of-scope)admin.example.com/panel(path)admin.example.com.(trailing dot)admin.example.com,(trailing comma)https://*.corp.example.com2001:db8::/32(IPv6 CIDR)re:^admin[.example\.com$(invalid regex)ValueError/scope.pyexits 2#155 reduced URL bullets to their host, but only for
scope.mdin-scope/out-of-scope bullets. Rules fromscope.json,--in-scope/--out-of-scope, and bare tokens with a port or path were not normalized.The fix
Scope.__init__runs every rule through_normalize_rule, whatever its source (scope.md,scope.json, CLI flags, the osint/recon helpers). It reduces the rule to its host: it strips the scheme, userinfo, port, path, trailing dot, wrapping backticks/quotes and trailing,/;, and lowercases it.*.wildcards andre:rules are kept as they are. The md-only_rule_from_tokenfrom fix(scope): reduce URL bullets in scope.md to their host #155 is now the general rule and is removed.2001:db8::/32,[2001:db8::1]) and compared in canonical form (2001:0db8::0001==2001:db8::1). An IPv4 CIDR never matches an IPv6 host, and an IP rule never matches a hostname.re:rule that does not compile raisesValueErrorinstead of being skipped. An invalid in-scope regex already fails closed, so it only produces a warning.api.*.example.com, whitespace), the problem is recorded inScope.warnings.scope.pyprints these to stderr and the engine writes them to its log at start.Scope.loadreadsscope.jsonas UTF-8, matching the Windows fixes in fix(engine,cli): cross-platform Windows support, utf-8 file writing, and triage test suite #160.Behaviour for rules that already worked is unchanged. All existing scope tests and the
--selftestpass without edits.Type
Checklist
example.com/ RFC 38492001:db8::/32are used.python3 scripts/lint_skills.pypasses locally (no skill changes);scripts/scan_identifiers.py: 0 matches;scripts/check_doc_counts.py: 0 errors.engine/test_scope_rule_normalization.py(14 tests, 16 failures + 3 errors onmain, all pass here). The fullengine/suite passes, as dotests/and every step ofruntime-tests.yml(self-tests, mock engine run,cbh --help). Code is Python 3.9-compatible.scope.pydocstring,commands/scope.mdrule list,CHANGELOG.md.Anything reviewers should know?
admin.example.com/panelcan't be enforced at path level. Before this PR it was ignored and/panelgot tested. Denying the host is the conservative choice for a deny-wins gate, and the warning tells the user what happened. Path-level enforcement would be a bigger change and could be a follow-up if you want it.scope.mdURL bullets, and a warning is printed.http://2130706433/,0x7f.1). Those already fail closed for in-scope rules.CHANGELOG.md.