Skip to content

fix(scope): stop out-of-scope rules from silently failing open - #182

Open
fairoz-539 wants to merge 2 commits into
elementalsouls:mainfrom
fairoz-539:fix/scope-deny-rules-fail-open
Open

fairoz-539 wants to merge 2 commits into
elementalsouls:mainfrom
fairoz-539:fix/scope-deny-rules-fail-open

Conversation

@fairoz-539

Copy link
Copy Markdown

What does this change?

engine/scope.py is the deterministic gate the engine and /scope rely on ("deny wins, default deny"). Several common ways of writing an out-of-scope asset match no host. The deny rule is skipped without any error, so the host falls through to a broader in-scope rule and gets tested.

Repro on main (Scope(in_scope=["example.com"], out_of_scope=[RULE]), target https://admin.example.com/):

out-of-scope rule as written main this PR
admin.example.com:8443 (port, common on program pages) IN scope denied
https://admin.example.com (scheme, scope.json / --out-of-scope) IN scope denied
admin.example.com/panel (path) IN scope denied (whole host, warned)
admin.example.com. (trailing dot) IN scope denied
admin.example.com, (trailing comma) IN scope denied
https://*.corp.example.com IN scope denied
2001:db8::/32 (IPv6 CIDR) IN scope denied
re:^admin[.example\.com$ (invalid regex) IN scope, no error ValueError / scope.py exits 2
$ python3 engine/scope.py admin.example.com --in-scope example.com --out-of-scope 're:(unclosed'
IN-SCOPE   admin.example.com          # main
scope error: invalid out-of-scope regex 're:(unclosed': ... refusing to build a scope whose deny rule cannot be enforced   # this PR, exit 2

#155 reduced URL bullets to their host, but only for scope.md in-scope/out-of-scope bullets. Rules from scope.json, --in-scope/--out-of-scope, and bare tokens with a port or path were not normalized.

The fix

  • One normalization point: Scope.__init__ runs every rule through _normalize_rule, whatever its source (scope.md, scope.json, CLI flags, the osint/recon helpers). It reduces the rule to its host: it strips the scheme, userinfo, port, path, trailing dot, wrapping backticks/quotes and trailing ,/;, and lowercases it. *. wildcards and re: rules are kept as they are. The md-only _rule_from_token from fix(scope): reduce URL bullets in scope.md to their host #155 is now the general rule and is removed.
  • IPv6: addresses and CIDRs are supported (2001:db8::/32, [2001:db8::1]) and compared in canonical form (2001:0db8::0001 == 2001:db8::1). An IPv4 CIDR never matches an IPv6 host, and an IP rule never matches a hostname.
  • Fail closed on an unenforceable deny: an out-of-scope re: rule that does not compile raises ValueError instead of being skipped. An invalid in-scope regex already fails closed, so it only produces a warning.
  • Visible, not silent: when a rule is widened to its whole host (it had a path or port, which can't be enforced per host) or can never match (api.*.example.com, whitespace), the problem is recorded in Scope.warnings. scope.py prints these to stderr and the engine writes them to its log at start.
  • Scope.load reads scope.json as UTF-8, matching the Windows fixes in fix(engine,cli): cross-platform Windows support, utf-8 file writing, and triage test suite #160.

Behaviour for rules that already worked is unchanged. All existing scope tests and the --selftest pass without edits.

Type

  • New skill
  • Improvement to an existing skill
  • Docs / examples
  • Tooling / scripts / CI
  • Other:

Checklist

  • No client/engagement data. Only example.com / RFC 3849 2001:db8::/32 are used.
  • python3 scripts/lint_skills.py passes locally (no skill changes); scripts/scan_identifiers.py: 0 matches; scripts/check_doc_counts.py: 0 errors.
  • Tests: new engine/test_scope_rule_normalization.py (14 tests, 16 failures + 3 errors on main, all pass here). The full engine/ suite passes, as do tests/ and every step of runtime-tests.yml (self-tests, mock engine run, cbh --help). Code is Python 3.9-compatible.
  • Docs: scope.py docstring, commands/scope.md rule list, CHANGELOG.md.

Anything reviewers should know?

  • Path-scoped out-of-scope rules now deny the whole host. Scope is enforced per host, so admin.example.com/panel can't be enforced at path level. Before this PR it was ignored and /panel got tested. Denying the host is the conservative choice for a deny-wins gate, and the warning tells the user what happened. Path-level enforcement would be a bigger change and could be a follow-up if you want it.
  • In-scope rules with a path/port are widened to the host. That matches what fix(scope): reduce URL bullets in scope.md to their host #155 already does for scope.md URL bullets, and a warning is printed.
  • Out of scope for this PR: non-canonical IPv4 forms (http://2130706433/, 0x7f.1). Those already fail closed for in-scope rules.
  • This PR is independent of the recon redirect PR. The two only touch the same lines in CHANGELOG.md.

A deny rule only works if it can match the host it names. Several common
ways of writing an out-of-scope asset matched nothing, so the host fell
through to a broader in-scope rule and the engine tested it:

  out_of_scope: admin.example.com:8443        (port)
  out_of_scope: https://admin.example.com     (scheme, scope.json / CLI)
  out_of_scope: admin.example.com/panel       (path)
  out_of_scope: admin.example.com.            (trailing dot)
  out_of_scope: 2001:db8::/32                 (IPv6 CIDR)
  out_of_scope: re:^admin[.example\.com$      (invalid regex -> skipped)

Normalize every rule once, in Scope, whatever its source (scope.md,
scope.json, --in-scope/--out-of-scope, osint/recon helpers): reduce it
to its host, canonicalize IPv4/IPv6 addresses and CIDRs, and record a
warning when a path or port is dropped or a rule still cannot match.
An out-of-scope regex that does not compile now raises ValueError (exit
2 from scope.py) instead of being skipped. Warnings are printed by
scope.py and written to the engine log. scope.json is read as UTF-8.

The scope.md-only URL reduction from elementalsouls#155 is now the general rule, so
_rule_from_token is removed.

@auditbenchaibot auditbenchaibot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open in Audit Bench Ai

🤖 Automated review by Audit Bench Ai, not a personal review. It was posted through the GitHub account connected to Audit Bench Ai. Install the Audit Bench Ai GitHub App to have reviews posted by a separate bot account.

No actionable comments were generated in this review. 🎉

Merge risk: 🟢 Low · up to 483ed60

Targets previously matched by lenient in-scope fallthrough may now be blocked or cause errors if invalid deny regexes are provided, but this strictly enforces intended safety boundaries.

Merge readiness: 🟡 Ready to merge, with notes

Pre-merge checks: ✅ 8 · ⚠️ 2

⚠️ Checks to review: 2 warnings

Check name Status Explanation Resolution
CI checks ⚠️ No CI checks ran on this commit. Add CI so every PR is built and tested.
Tests ⚠️ 3 source files changed but no test files did.
📄 engine/scope.py
📄 engine/engine.py
📄 engine/test_scope_rule_normalization.py
Add or update tests that cover the new behaviour.
✅ Passed checks: 8 passed
Check name Status Explanation
Merge conflicts ✅ Merges cleanly into main.
Secrets ✅ No keys, passwords or tokens in the changed code.
Blocking issues ✅ No critical or high-severity issues in the changed lines.
PR size ✅ 305 lines changed.
Title check ✅ The title clearly states that out-of-scope rules are prevented from failing open.
Description check ✅ The description provides detailed context, reproduction tables, and an explanation of the fix.
Linked issues check ✅ Mentions related PRs #155 and #160 as background context, and the implemented changes directly follow that lineage without unresolved issue links.
Out of scope changes check ✅ All modifications in the engine, scope module, and test suite directly relate to scope rule normalization and enforcement.
📝 Walkthrough

This pull request fixes an issue where out-of-scope rules with ports, paths, schemes, or formatting quirks failed to match and silently allowed targets to remain in-scope. It introduces centralized rule normalization in Scope.init, adds IPv6 CIDR and canonical address matching, and raises a ValueError on uncompilable out-of-scope regex rules to fail closed. In addition, rule widening or invalid patterns now generate warnings logged by the engine and CLI.

File Changes
engine/engine.py Logs warnings emitted during scope normalization on engine startup.
engine/scope.py Implements centralized rule normalization, canonical IP/IPv6 CIDR handling, fail-closed deny regex parsing, and warning collection.
engine/test_scope_rule_normalization.py Adds comprehensive tests verifying deny rule forms, IPv6 matching, regex error handling, and scope warnings.
ℹ️ Review info
  • Commit: 483ed60
  • Verdict: ✅ Pass
  • Findings: none
  • Review: local checks plus AI review of the changed lines
  • Scope: Reviewed 3/3 selected files (of 3 total): 3 reviewed by AI, 0 from cache, 0 passed local checks with nothing risky enough to send to AI. Found 0 finding(s), No secrets found.

💬 Reply to any inline comment and Audit Bench Ai will recheck that code and answer in the thread. Mention @auditbench in the conversation to ask about the whole PR.

🛠️ Open the full report and fix with AI

Thanks for using Audit Bench Ai! Free pull request reviews for open source: paste any public PR at auditbenchai.com/scan.

Reviewed by Audit Bench Ai · Build fast. Check what ships. · Review any public PR free

Resolve the engine/scope.py conflict with elementalsouls#191: both sides read
scope.json as UTF-8; keep the with-block form so the file is closed.
Drop the now-redundant UTF-8 note from the CHANGELOG entry.

@auditbenchaibot auditbenchaibot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open in Audit Bench Ai

🤖 Automated review by Audit Bench Ai, not a personal review. It was posted through the GitHub account connected to Audit Bench Ai. Install the Audit Bench Ai GitHub App to have reviews posted by a separate bot account.

Actionable comments posted: 1

Merge risk: 🟡 Medium · up to 3595415

URL-formatted IP rules with subnets may incorrectly strip netmasks, and regex matching failures at runtime could bypass deny protections unexpectedly.

Merge readiness: ⛔ Not ready to merge

Pre-merge checks: ✅ 7 · ⚠️ 2 · ❌ 1

❌ Failed checks: 1 error, 2 warnings

Check name Status Explanation Resolution
CI checks ⚠️ No CI checks ran on this commit. Add CI so every PR is built and tested.
Blocking issues ❌ 1 critical issue in the changed lines.
📄 engine/scope.py:99
Fix the critical inline comments before merging.
Tests ⚠️ 3 source files changed but no test files did.
📄 engine/scope.py
📄 engine/engine.py
📄 engine/test_scope_rule_normalization.py
Add or update tests that cover the new behaviour.
✅ Passed checks: 7 passed
Check name Status Explanation
Merge conflicts ✅ Merges cleanly into main.
Secrets ✅ No keys, passwords or tokens in the changed code.
PR size ✅ 305 lines changed.
Title check ✅ The title clearly states that out-of-scope rules are being fixed to stop failing open.
Description check ✅ The description provides comprehensive background, reproduction examples, and details of the changes.
Linked issues check ✅ References #155 and #160 for background context on past normalization and encoding fixes, which align with this implementation.
Out of scope changes check ✅ All changes directly pertain to scope parsing, normalization, warning logs, and relevant unit tests.
📝 Walkthrough

This pull request ensures scope rules (especially out-of-scope deny rules) are consistently normalized regardless of input format or source. Rules specifying schemes, ports, paths, or IPv6 formats are mapped to canonical host or CIDR patterns rather than silently failing open. Additionally, invalid deny regex patterns now fail closed by raising an error instead of being skipped.

File Changes
engine/engine.py Logs scope normalization warnings when the engine starts.
engine/scope.py Adds universal rule normalization, IPv6 CIDR support, and fails closed on invalid deny regexes.
engine/test_scope_rule_normalization.py Adds comprehensive unit tests verifying scope rule normalization, regex failure modes, and warning behavior.
ℹ️ Review info
  • Commit: 3595415
  • Verdict: ⚠️ Needs work
  • Findings: 1 critical, 1 high
  • Review: local checks plus AI review of the changed lines
  • Scope: Reviewed 3/3 selected files (of 3 total): 2 reviewed by AI, 1 from cache, 0 passed local checks with nothing risky enough to send to AI. Found 2 finding(s), No secrets found.
🤖 Prompt to fix review comments
Verify each finding against the current code and only fix it if it still applies.

In `@engine/scope.py` around lines 99 - 103: Invalid or uncompilable out-of-scope regex fails open instead of blocking targets. When matching a pattern in `_match`, if a regex pattern (`re:...`) fails to compile or throws an error during `re.search`, the exception is swallowed and returns `False`. If this occurs during the evaluation of an out-of-scope rule in `in_scope_host` (`any(_match(p, host) for p in self.out_of_scope)`), returning `False` means the deny rule does not match, allowing the target to be considered in-scope. This violates the core design principle that deny rules must never fail open. Suggested fix: Let `re.error` propagate or treat invalid regex evaluation in deny rules as matching (or failing closed) so that scanning agents are not inadvertently permitted to access targets when an out-of-scope regex fails.

In `@engine/scope.py` around lines 74 - 77: Subnet masks in URL-formatted rules are stripped into single host IPs. When an IP CIDR block is supplied in URL format (such as `http://10.0.0.0/24` or `10.0.0.0/24`), `_canonical_ip(t)` on line 71 handles bare CIDRs like `10.0.0.0/24`. However, if the rule has a scheme like `http://10.0.0.0/24` or extra URL components, it bypasses `_canonical_ip(t)` and is parsed via `urlparse('//' + t)`. In that case, `urlparse` treats `/24` as `parsed.path` and `parsed.hostname` becomes just `10.0.0.0`. The rule is normalized to the single IP `10.0.0.0`, discarding the `/24` subnet entirely and leaving the rest of the subnet unprotected by out-of-scope deny rules. Suggested fix: Before parsing with `urlparse`, check if the host component contains an IP network CIDR, or re-attach the path if the host is an IP address and `parsed.path` starts with a valid network prefix length.

💬 Reply to any inline comment and Audit Bench Ai will recheck that code and answer in the thread. Mention @auditbench in the conversation to ask about the whole PR.

🛠️ Open the full report and fix with AI

Thanks for using Audit Bench Ai! Free pull request reviews for open source: paste any public PR at auditbenchai.com/scan.

Reviewed by Audit Bench Ai · Build fast. Check what ships. · Review any public PR free

Comment thread engine/scope.py
Comment on lines +74 to +77
try:
parsed = urlparse(t if "://" in t or t.startswith("//") else "//" + t)
host = (parsed.hostname or "").rstrip(".")
port = parsed.port

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Audit Bench Ai (automated) | 🧠 Logic | 🟠 High

Subnet masks in URL-formatted rules are stripped into single host IPs

When an IP CIDR block is supplied in URL format (such as http://10.0.0.0/24 or 10.0.0.0/24), _canonical_ip(t) on line 71 handles bare CIDRs like 10.0.0.0/24. However, if the rule has a scheme like http://10.0.0.0/24 or extra URL components, it bypasses _canonical_ip(t) and is parsed via urlparse('//' + t). In that case, urlparse treats /24 as parsed.path and parsed.hostname becomes just 10.0.0.0. The rule is normalized to the single IP 10.0.0.0, discarding the /24 subnet entirely and leaving the rest of the subnet unprotected by out-of-scope deny rules.

Why it happens: urlparse splits the CIDR prefix into the path component when parsing URLs containing CIDRs, and _normalize_rule assigns host = parsed.hostname.

When it breaks: An out-of-scope rule is entered as https://192.168.1.0/24. Normalization reduces the rule to 192.168.1.0. A target at 192.168.1.5 is checked; it does not match 192.168.1.0, bypassing the out-of-scope restriction entirely.

Tests: Test test_regex_and_cidr_rules_are_untouched only tests bare CIDRs like 2001:db8::/32, not CIDRs entered with a scheme or URL prefix.

Suggested fix: Before parsing with urlparse, check if the host component contains an IP network CIDR, or re-attach the path if the host is an IP address and parsed.path starts with a valid network prefix length.

95% confidence · 💬 Reply here and Audit Bench Ai will recheck the code · 👀 to acknowledge · 👍/👎 teaches it what your team wants

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants