Decompose utils.py: Ca=19 hub drives the dependency-depth and coupling scores

[thodson-usgs]

The weekly pyscn sweep added in #360 scores the package 78/100 (B). Two sub-scores carry almost the whole deficit, and both trace back to one module. Filing the analysis so the work is scoped rather than rediscovered.

utils.py is the hub

metric	value
afferent coupling (Ca)	19 — the highest in the package; next is 11
lines	528
responsibilities	metadata, frame shaping, ambient config, legacy request composition, transport compatibility

The architecture overview already lists this under Known architectural debt: "utils.py combines metadata, shaping, ambient configuration, legacy request composition, and transport compatibility imports." This is the measured version of that note.

The cost is concrete. Because utils imports the HTTP stack (transport.http, transport.retry, credentials, exceptions, codes), anything that needs one helper from it inherits all of that transitively. The clearest case:

ogc/context.py is 20 lines. Its only substantive import is Ambient — a ContextVar wrapper that depends on nothing but contextvars and contextlib. Importing it from utils puts the entire HTTP stack in ogc.context's transitive closure, and lands it on the longest dependency chain in the package:

dataretrieval -> waterdata -> waterdata.nearest -> waterdata.time_series
  -> waterdata.stats -> waterdata.utils -> ogc -> ogc.engine -> ogc.shaping
  -> ogc.schema -> ogc.context -> utils -> transport.retry -> progress -> credentials

Note Ambient is not used by utils itself — only by ogc.context and ogc.chunking. It lives there by accident.

Suggested first step

Move Ambient to a dependency-free leaf module (dataretrieval/ambient.py). Small, self-contained, two import sites.

Measured honestly: this alone moves max graph depth 14 → 13, because ogc.schema -> ogc.errors -> transport.pagination is a nearly-as-long parallel path. So do it for the coupling and the clarity, not for the score — the number barely moves until the wider decomposition happens. Adding it to .importlinter's layer stack is required (the stack is exhaustive), which is the contract doing its job.

The wider work

Split utils.py along the seams the architecture doc already names, so consumers depend on what they use:

• ambient/context helpers (stdlib-only leaf)
• validation helpers (_require_positive_int and friends)
• frame/metadata shaping
• legacy query composition (used by wqp, nldi, streamstats, nwis)

Each move is independently testable and each should shrink utils's Ca. The .importlinter contracts make the direction checkable at every step.

What is deliberately not worth doing

Recording these so nobody "fixes" the score later by making the library worse:

• Duplication 60/100. All 8 clone groups are false positives: the per-collection getter families (thin, heavily documented wrappers whose bodies necessarily rhyme) and deliberate pairs like utils.query/_query_with_retry and ChunkCall.partial_frame/partial_response. Collapsing them trades the documented public surface for a metric.
• 19 "high-risk modules" / Modularity 50. Driven by abstractness = 0 across all 50 modules, so every stable leaf sits at distance 1 from the main sequence ("zone of pain"). Raising it means adding ABCs to exceptions, codes, rdb, and waterdata.types — textbook metric gaming, and against this project's stated preference for functions over classes.
• Architecture 85/100. The 25 "violations" are mostly an SRP heuristic flagging exceptions and credentials for being imported by many packages. That is afferent coupling on a stable leaf, which is exactly what ADR 0003 asks for. Its DI check also misidentifies a function (planning._extract_axes) as a class.
• "20 high-risk functions" (Complexity 95/100). 17 of the 20 are synthetic <module> rows where pyscn sums every function in a file — planning.py shows cognitive 140 with cyclomatic 1. Only 3 are real functions.

Related: laddering the complexity ratchet

#360 refactored ogc.filters._split_top_level_or (extracting the OR-token lookahead) and tightened the complexipy gate from 36 to 27. The next rungs, in order:

function	cognitive
nldi.get_features	27
ogc.requests._construct_api_requests	25
nwis._read_json	25 — deprecated module, leave alone
ogc.planning.ChunkPlan._refine	22

Each one that comes down lets the gate tighten by the same amount, permanently. Worth doing opportunistically when touching those functions rather than as a dedicated refactor.

[thodson-usgs]

wily history: the accretion is measurable, and partly addressed

Ran wily build over the last 120 revisions. The historical view says something a point-in-time scan cannot — utils.py isn't just large, it was actively growing while everything around it improved.

file	MI then → now	SLOC	cyclomatic
waterdata/api.py	6.8 → 100	375 → 51	−48
waterdata/utils.py	34.5 → 74.3	1069 → 363	−108
waterdata (pkg)	60.3 → 82.0	—	−33
nwis.py	28.7 → 39.4	511 → 454	−22
utils.py	66.6 → 56.1	234 → 527	+27

The Water Data split moved 706 lines out of waterdata/utils.py, and the shared utils.py more than doubled (5 → 14 functions) over the same window. Code left a module that was being watched and landed in one that wasn't.

Partly addressed in #360

Auditing what the dependents actually wanted: 15 of the 21 import sites wanted one name — BaseMetadata. A class whose only dependency is httpx.Response, sitting beside the legacy query machinery, so importing it pulled transport + credentials + error policy in transitively.

Extracted to dataretrieval/response_metadata.py (dependency-free leaf; dataretrieval.utils.BaseMetadata still resolves via re-export). Ca(utils): 20 → 10. Ten modules — ngwmn, ogc.shaping, and eight waterdata modules — no longer depend on utils at all.

Max depth stays 14; simulated before doing it. That number is driven by the legitimate waterdata → ogc → transport layering, not by this edge, so it is not a useful target.

What remains

utils.py still has Ca=10 and holds two unrelated halves:

• legacy HTTP query composition — query, _query_with_retry, _query_impl, _get_with_retry, _raise_for_status, _single_request_policy; used by wqp, nldi, nwis, streamstats. This half is what makes utils import transport.*, credentials, exceptions.
• pure stdlib helpers — Ambient, _require_positive_int, to_str; used only by OGC and two services, zero HTTP dependencies.

Splitting those is the rest of this issue. The pure half is mechanical. The legacy-query half is entangled with the NWIS deprecation, so it may be cheaper to wait for that removal than to move it twice.

An empirical caution about the score

The BaseMetadata extraction is unambiguously good design: a dependency-free leaf, coupling halved, public path preserved, all tests and contracts green.

pyscn's architecture score went 85 → 81 for it.

The new leaf is now itself an "SRP violation" with responsibilities = [ngwmn, nwis, ogc, waterdata, wateruse, wqp] — penalized for being depended upon — and every module that imports it gained a "responsibility". That is a controlled before/after in which a real improvement moves the metric backwards, and it is the clearest reason to keep pyscn advisory rather than a merge gate: as a gate with a threshold, it would have blocked this change.