diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 00000000..8b9e2032 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,164 @@ +# Context + +The shared vocabulary for `dataretrieval`. This is a glossary, not a +specification: it fixes what words mean so that code, docstrings, ADRs, and +conversation use them the same way. Architectural decisions live in +`docs/source/architecture/decisions/`. + +When a term here conflicts with a name in the code, the term wins and the name +is legacy. Legacy names are called out below rather than quietly tolerated. + +## Retrieval + +**Getter** — A public function that retrieves data and returns +`(DataFrame, metadata)`. The package's unit of public API. `waterdata.get_daily` +and `wqp.get_results` are getters; the helpers they call are not. + +**Query** — One logical request a caller makes by calling a getter. A query may +reach the service as several requests; it is still one query. + +**Chunk** — One of the requests a query was split into. Following Dask, where +chunks describe how an array is split into sub-arrays, a chunk is a piece of the +whole, named for *being a piece* rather than for why it was made one. + +A query needing no split has exactly one chunk, not zero. + +**Chunking** — How a query is split into chunks. A query may be chunked because +the service forces it — a URL over the byte limit, or an API accepting one +location per request — or because the caller asked for it. Both produce chunks; +the reason is not part of the term. + +**Plan** — An enumeration of a query's chunks: how many there are, and what each +one is. A plan says how a query divides; it does not execute. Computing a plan +is protocol-specific — a byte budget, a per-location rule — while executing one +is not, which is why the two live apart. + +**Fan-out** — Executing a query's chunks concurrently. Chunking is how the work +divides; fan-out is how it is distributed. The two are independent, and only +chunking depends on the service's protocol. + +**Page** — One response in a cursor-followed sequence from a single chunk. A +page is *not* a chunk: chunks divide a query, pages divide a chunk's response. +A chunk of a large query commonly spans many pages. + +## Failure and resumption + +**Transient failure** — A failure a later attempt could survive: a rate limit, a +service error, a timeout. Distinguished from a **deterministic failure**, which +would fail identically every time — an unresolvable hostname, an unsupported +scheme, a malformed request. Only transient failures are retried, and only +transient failures produce a resumable interruption. The two answers are one +judgement about what a failure means, and must agree. + +**Interruption** — A transient failure that stopped a fan-out partway, raised +with the completed chunks preserved. The caller may wait for the condition to +clear and resume. + +**Resume** — Continuing an interrupted query by re-issuing only the chunks that +did not complete. Completed chunks are never re-fetched. A chunk that failed +partway through its pages is not complete, so resuming re-walks its pages from +the start. + +## Services + +Each is an external system this package retrieves from. They are separate +services with separate conventions, not one API with modes. + +**Water Data** — The modern USGS API at `api.waterdata.usgs.gov`, covering +monitoring locations, time series, field measurements, samples, ratings, and +statistics. The package's primary target. + +**NGWMN** — The National Ground-Water Monitoring Network, a distinct OGC API +covering sites, water levels, lithology, well construction, and providers. + +**NWDC** — The National Water Availability Assessment Data Companion, providing +modeled national-scale water-use data. + +**WQP** — The Water Quality Portal, a multi-agency water-quality clearinghouse. + +**NLDI** — The Network Linked Data Index, which navigates the hydrologic network +from an origin to connected features, flowlines, or basins. + +**NWIS** — The legacy USGS waterservices interface. Deprecated: it is retained +for compatibility and is not where new work goes. + +**StreamStats** — Basin characteristics and delineation for a point on a stream. + +## Data + +**Collection** — One named set of records a service offers — `daily`, +`monitoring-locations`, `time-series-metadata`. The unit a getter targets. + +A collection is not a service. Water Data is a service; `daily` is one of its +collections. The distinction matters because the OGC machinery is shared: the +same code path retrieves a Water Data collection and an NGWMN one, and only the +service differs. + +**Collection family** — A group of collections sharing a shape and therefore a +getter signature. Their getters deliberately resemble one another; the +resemblance is the public contract, not duplication to be removed. + +**Monitoring location** — A place where measurements are recorded. The canonical +term. Legacy: the deprecated NWIS getters and the WQP profiles call this a +*site*, and their parameters keep that spelling. + +**Metadata** — The second half of every getter's return: the request URL, the +elapsed time, and the response headers. Describes the *retrieval*, not the data. + +## Boundaries + +**Adapter** — A module owning one service's conventions: its URLs, parameters, +error shapes, and response quirks. Adapters may use shared machinery; shared +machinery may not know about adapters. + +**Facade** — A module that re-exports a subsystem's public surface and contains +no logic of its own, so callers depend on a stable name rather than on internal +layout. + +**Leaf** — A module with no dependencies inside the package beyond other leaves, +holding one general mechanism so that anything may use it without acquiring the +rest of the package. Before writing a small helper, check whether a leaf already +generalizes it. + +**Transport** — The service-neutral machinery for issuing requests: timeouts, +retry, pagination, fan-out, aggregation. It names no service and no protocol, +and is not public API. + +## Known legacy names + +Recorded so they are not mistaken for the canonical term, and not re-litigated: + +- `completed_chunks` / `total_chunks` on interruptions, and `set_chunks()` / + `start_chunk()` on the progress reporter, count chunks as defined above and + are consistent with this glossary. They predate it; the agreement is real + rather than coincidental. +- `ChunkInterrupted` is a permanent alias of `FanOutInterrupted` — the same + class object under the name it was first published as. Both spellings are + correct; neither is scheduled for removal. +- `site` appears in deprecated NWIS and WQP parameter names where *monitoring + location* is meant. These are frozen public surfaces and will not be renamed. + Where the Water Data API itself names a thing `site-types` or + `site_type_code`, that is the service's vocabulary and is reproduced + faithfully rather than translated. +- `service` named a collection throughout the OGC machinery. Resolved: the + OGC internals, the Water Data wrappers, and all eleven typed getters now say + `collection`; `waterdata.get_cql` takes `collection`; and the type alias is + `WATERDATA_COLLECTIONS`. `service=` on `get_cql` and the `WATERDATA_SERVICES` + alias remain — a deprecated keyword and a permanent alias respectively. + + `service` still means the external system in `transport` and `progress`, + where it labels a progress line. That usage is correct. +- `waterdata.get_samples(service=)` names a *resource*, not a service. The + Samples OpenAPI document declares `results`, `locations`, `activities`, + `projects` and `organizations` as tags and titles itself the "Resource + Center"; it never calls them services. They are also not collections -- + they share 22 of 23 query parameters, so they are five projections of one + query rather than five sets of data, and the OGC definition of *collection* + is scoped to "access mechanisms defined by OGC API standard(s)", which + Samples does not implement. Kept as-is by decision: renaming a public + keyword costs a deprecation cycle for a term with no better-evidenced + replacement in reach. +- `waterdata.get_codes(code_service=)` is correct and stays. The Samples + documentation calls it a "code service" in prose and serves it from + `/codeservice/`, so this reproduces the service's own vocabulary, like + `site-types`. diff --git a/NEWS.md b/NEWS.md index bb5ae8cd..2f1b52d2 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,5 @@ +**08/09/2026:** `waterdata.get_cql` takes `collection` rather than `service`. OGC API - Features (17-069r4) normatively names this value the `collectionId`: Requirement 20 fixes the path template `/collections/{collectionId}/items`, and Requirement 18 defines `collectionId` as each `id` in the collections response -- which is literally how the package builds the URL, and what the live API returns. *Service* names the API itself (Water Data, NGWMN). **Deprecation:** `service=` still works and resolves to `collection`, with a `DeprecationWarning`; it will be removed on or after 2027-08-09. Positional callers (`get_cql("daily", cql)`) are unaffected. The `WATERDATA_SERVICES` type alias is now `WATERDATA_COLLECTIONS`, with `WATERDATA_SERVICES` retained as a permanent alias for the same object. Terms are defined in `CONTEXT.md`. + **08/09/2026:** Every retrieval path now runs through one executor. `waterdata.get_cql` (via the OGC `fetch_ogc_request`) and `waterdata.get_stats_por` / `get_stats_date_range` (via the Statistics page walk) previously bypassed `dataretrieval.transport.fanout.FanOut` through a private sync bridge, which meant they were the only getters in the package with **no retry**: a mid-page-walk 429 or 503 failed the whole call while every typed getter and Water Use rode it out. Both now run as a one-item fan-out and the 25-line `transport/sync.py` is gone. **Behavior change:** those three getters now retry transient failures (`API_USGS_RETRIES`, default 4) and, when the retries are exhausted, raise the resumable `ServiceInterrupted` / `QuotaExhausted` rather than `ServiceUnavailable` / `RateLimited` / `NetworkError` — all remain `DataRetrievalError`, so broad handlers are unaffected, but narrow handlers around those calls must widen, and `.call.resume()` is now available on the interruption. A failure that retrying cannot fix (bad scheme, a hostname that does not resolve) still surfaces as `NetworkError` immediately. The progress line moved with it: `FanOut.resume()` opens the reporter it ticks into, so a driver can no longer run the shared executor and silently print nothing, and a `.call.resume()` fired long after the interruption now reports progress instead of running mute. Internal tidying with no public effect: the WQX3 / legacy-WQP CSV datetime shaping moved out of `dataretrieval.utils` (whose docstring reserves it for non-service-specific shaping) into the `dataretrieval._wqx` leaf; the five Water Data endpoint URLs are declared once in `dataretrieval.waterdata.endpoints` instead of being derived in three modules; the OGC queryables document is parsed by `dataretrieval.ogc.schema` so every OGC adapter can offer the table, with `waterdata.get_queryables` unchanged as its documented wrapper; and `ogc/engine.py` imports each symbol from the module that defines it. **08/09/2026:** Internal structure cleanup, no public API change. Validating a server-supplied next-page link is now one policy in `dataretrieval.transport.links` instead of three divergent copies (the OGC engine, the ratings STAC walk, and Water Use). Two of those copies were fixed by the merge: the OGC page walk now resolves a *relative* `next` href against the page it came from (it previously handed the unresolved reference back as the pagination cursor) and refuses an unparseable one rather than following it unchecked. Cross-host refusal, credential stripping, and Water Use's host-alias rewrite are unchanged, as is the error type each walk raises. `parse_retry_after` moved to `dataretrieval.exceptions`, next to the `DataRetrievalError.retry_after` field it exists to produce. The one-shot HTTP query path (`query`, `to_str`, and their helpers) moved out of `dataretrieval.utils` into the private `dataretrieval._querying`; `dataretrieval.utils.query` and `dataretrieval.utils.to_str` remain the documented public paths, as `Ambient` and `BaseMetadata` already do. `waterdata` profile validation moved next to the tables it validates in `waterdata.types`, and `nwis.get_dv`/`get_iv` now share one body. diff --git a/dataretrieval/combining.py b/dataretrieval/combining.py index 6be92760..d052ea71 100644 --- a/dataretrieval/combining.py +++ b/dataretrieval/combining.py @@ -1,7 +1,7 @@ """Result recombination: merge per-chunk frames and responses (no I/O). These utilities assemble the output of a chunked/fan-out call from its -individual per-sub-request results. They have no event-loop, retry, or +individual per-chunk results. They have no event-loop, retry, or network state — they're pure data transforms shared by protocol-specific chunk execution, service fan-out, and cursor-driven pagination. @@ -157,7 +157,7 @@ def _combine_chunk_responses( responses: list[httpx.Response], canonical_url: str | None ) -> httpx.Response: """ - Fold per-sub-request responses into a single aggregated response. + Fold per-chunk responses into a single aggregated response. For a multi-response input, returns a shallow copy of ``responses[0]`` with ``.headers`` set to those of the response reporting @@ -174,7 +174,7 @@ def _combine_chunk_responses( Parameters ---------- responses : list[httpx.Response] - One response per completed sub-request, in caller-provided order. + One response per completed chunk, in caller-provided order. canonical_url : str or None URL of the unchunked original request. ``None`` skips the URL override — used by the passthrough path (the fetcher's diff --git a/dataretrieval/exceptions.py b/dataretrieval/exceptions.py index 99a77be3..79160e9e 100644 --- a/dataretrieval/exceptions.py +++ b/dataretrieval/exceptions.py @@ -227,8 +227,8 @@ class Unchunkable(RequestTooLarge): """No chunking plan fits the URL byte limit. Raised by the Water Data chunker when even the smallest reducible plan - (every list axis at one atom per sub-request, the filter at one clause per - sub-request) still exceeds the server's byte limit. Unlike + (every list axis at one atom per chunk, the filter at one clause per + chunk) still exceeds the server's byte limit. Unlike :class:`URLTooLong`, then, automatic splitting has already been tried and exhausted. Shrink the input lists, simplify the filter, or split the call manually. diff --git a/dataretrieval/interruptions.py b/dataretrieval/interruptions.py index 2d408ab7..0bd0d0cc 100644 --- a/dataretrieval/interruptions.py +++ b/dataretrieval/interruptions.py @@ -3,17 +3,18 @@ When a fanned-out request fails mid-stream (a 429, a 5xx, or a bare transport error), the work already completed is preserved and the call is resumable: the raised exception carries a ``.call`` handle whose ``resume()`` re-issues only -the still-pending sub-requests. These exception types are that contract, +the still-pending chunks. These exception types are that contract, re-exported at the top level (``from dataretrieval import ChunkInterrupted``). The execution machinery that raises and resumes them is :class:`dataretrieval.transport.fanout.FanOut`. -Vocabulary, consistently: a **fan-out** is one logical query the service forces -into several requests; a **sub-request** is one unit of a fan-out; a **chunk** -is specifically a *byte-driven* slice, which is OGC planning vocabulary and -belongs to :class:`~dataretrieval.ogc.planning.ChunkPlan`. Water Use fans out -without chunking anything — the NWDC simply accepts one location per request — -so the base class is :class:`FanOutInterrupted`. +Vocabulary, consistently (see ``CONTEXT.md``): a **chunk** is one of the +requests a query was split into, named for being a piece rather than for why it +became one; **chunking** is how a query is split; and a **fan-out** is the +concurrent execution of a query's chunks. Water Use chunks one request per +location, Water Data chunks to fit a URL byte budget -- different reasons, the +same word. The base class is named :class:`FanOutInterrupted` because the +failure interrupts the *execution*, not the split. ``ChunkInterrupted`` is retained as an alias of that same class, not a deprecated shim to delete later: it is the name published in the user guide and @@ -45,13 +46,13 @@ class FanOutInterrupted(DataRetrievalError): """ - Base class for mid-stream sub-request failures whose completed work + Base class for mid-stream chunk failures whose completed work is preserved and resumable. - A ``FanOutInterrupted`` subclass means: a sub-request failed, but + A ``FanOutInterrupted`` subclass means: a chunk failed, but ``FanOut`` still owns whatever completed successfully before the failure. Call ``self.call.resume()`` to pick up where the - failure stopped you — only still-pending sub-requests are + failure stopped you — only still-pending chunks are re-issued. Subclasses describe *why* ``FanOut`` stopped so callers can @@ -72,16 +73,16 @@ class FanOutInterrupted(DataRetrievalError): Seconds the server suggested waiting (``Retry-After`` header). ``None`` when the server gave no hint. completed_chunks : int - Number of sub-requests successfully completed before the failure. + Number of chunks successfully completed before the failure. total_chunks : int - Total sub-requests in the plan. + Total chunks in the plan. partial_frame : pandas.DataFrame Combined frame of work completed by the moment this exception was raised. Snapshot at raise time — does NOT advance on a later ``call.resume()`` (use ``exc.call.partial_frame`` for the live view). partial_response : httpx.Response or None - Raw aggregate response covering the completed sub-requests at + Raw aggregate response covering the completed chunks at raise time; ``None`` if nothing had completed yet. Same snapshot semantics as ``partial_frame``. (Raw, not finalized — use ``exc.call.resume()`` for the finalized ``(df, metadata)`` result.) @@ -91,7 +92,7 @@ class FanOutInterrupted(DataRetrievalError): Retry on any transient interruption, honoring the server's ``Retry-After`` hint when present and falling back to a fixed wait otherwise. Each new interruption keeps the already-completed work - intact — only the still-pending sub-requests are re-issued. + intact — only the still-pending chunks are re-issued. .. code-block:: python @@ -116,7 +117,7 @@ class FanOutInterrupted(DataRetrievalError): # call sees ``completed_chunks`` and ``total_chunks`` as kwargs. _MESSAGE_TEMPLATE: ClassVar[str] = ( "Fan-out interrupted after {completed_chunks}/" - "{total_chunks} sub-requests; call .call.resume() to continue." + "{total_chunks} chunks; call .call.resume() to continue." ) retryable: ClassVar[bool] = True @@ -177,17 +178,17 @@ def __getstate__(self) -> dict[str, Any]: class QuotaExhausted(FanOutInterrupted): """ - A sub-request returned HTTP 429 — the per-key rate-limit window + A chunk returned HTTP 429 — the per-key rate-limit window is exhausted. Subclass of :class:`FanOutInterrupted`. - The completed sub-requests are preserved on ``.call``; once the + The completed chunks are preserved on ``.call``; once the rate-limit window resets, ``.call.resume()`` re-issues only the still-pending work. ``partial_frame`` holds what completed before the 429. """ _MESSAGE_TEMPLATE = ( - "HTTP 429 after {completed_chunks}/{total_chunks} sub-requests; " + "HTTP 429 after {completed_chunks}/{total_chunks} chunks; " "catch QuotaExhausted (or FanOutInterrupted) to access " ".partial_frame or .call.resume() once the rate-limit " "window has rolled over." @@ -197,17 +198,17 @@ class QuotaExhausted(FanOutInterrupted): class ServiceInterrupted(FanOutInterrupted): """ - A sub-request returned HTTP 5xx — the upstream service failed + A chunk returned HTTP 5xx — the upstream service failed transiently. Subclass of :class:`FanOutInterrupted`. - The completed sub-requests are preserved on ``.call``; once the + The completed chunks are preserved on ``.call``; once the upstream recovers, ``.call.resume()`` resumes only the still-pending work. """ _MESSAGE_TEMPLATE = ( "Service error after {completed_chunks}/{total_chunks} " - "sub-requests; catch ServiceInterrupted (or FanOutInterrupted) " + "chunks; catch ServiceInterrupted (or FanOutInterrupted) " "and call .call.resume() once the upstream service recovers." ) diff --git a/dataretrieval/ogc/__init__.py b/dataretrieval/ogc/__init__.py index ab3502cd..4a091e32 100644 --- a/dataretrieval/ogc/__init__.py +++ b/dataretrieval/ogc/__init__.py @@ -1,13 +1,13 @@ """Generic OGC API engine shared by the Water Data and NGWMN getters. -The public facade exposes only the minimal service-adapter seam: +The public facade exposes only the minimal collection-adapter seam: - :class:`OgcDialect` — per-API request/response quirks. - :func:`prepare_request_args` — normalize caller kwargs for the engine. - :func:`get_ogc_data` — full orchestrated OGC fetch (chunking + pagination). - :func:`fetch_ogc_request` — execute a pre-built request with pagination. -Service adapters (NGWMN, Water Data's generic wrapper) import from this +Collection adapters (NGWMN, Water Data's generic wrapper) import from this facade rather than reaching into engine internals. Generic execution policy lives in :mod:`dataretrieval.transport`; the engine retains compatibility wrappers at previous private paths. diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 7aad9165..c958b49d 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -4,13 +4,13 @@ parameter (sites, parameter codes, …) plus the cql-text ``filter``, which splits along its top-level OR clauses. Any of them can fan the URL past the server's ~8 KB byte limit. ``ChunkPlan`` picks a fan-out -for each axis that minimizes total sub-requests while keeping every -sub-request URL under the budget. Requests that already fit get a +for each axis that minimizes total chunks while keeping every +chunk URL under the budget. Requests that already fit get a trivial single-step plan — the executor has one code path either way. This module owns the OGC-specific half: the byte budget, the ``parallel_chunks`` dial, and the ``multi_value_chunked`` decorator that -ties a plan to a fetcher. Driving the resulting sub-requests to +ties a plan to a fetcher. Driving the resulting chunks to completion — bounded concurrency, retry, failure precedence, resume — is API-neutral and belongs to :class:`dataretrieval.transport.fanout.FanOut`, which this module hands @@ -20,7 +20,7 @@ Parallel chunks: the planner is conservative by default — it splits only as far as the byte limit forces. A caller who knows their result is large can opt into a finer split via the ``parallel_chunks(n)`` context manager, which fans -the query out into ``n`` parallel sub-requests. ``n`` drives +the query out into ``n`` parallel chunks. ``n`` drives :meth:`ChunkPlan._refine`; see ``parallel_chunks`` for the why and the when. Concurrency, retries, and interruption semantics are documented on @@ -82,7 +82,7 @@ # limit alone requires. Scoped to a ``with parallel_chunks(...):`` block (a # ContextVar), deliberately NOT an env var (see :func:`parallel_chunks` for # why). The ambient holds ``n`` — the requested cap on the plan's total -# sub-request count; ``1`` (the default, outside any block) means "off — chunk +# chunk count; ``1`` (the default, outside any block) means "off — chunk # only as much as the byte limit needs, no extra fan-out". _parallel_chunks: Ambient[int] = Ambient("ogc_parallel_chunks", 1) @@ -90,19 +90,19 @@ @contextmanager def parallel_chunks(n: int) -> Iterator[None]: """ - Fan the OGC getters' multi-value requests out into ``n`` parallel sub-requests. + Fan the OGC getters' multi-value requests out into ``n`` parallel chunks. By default the Water Data / NGWMN getters chunk a request only as much as - the server's ~8 KB URL-byte limit forces — the fewest sub-requests that + the server's ~8 KB URL-byte limit forces — the fewest chunks that fit. That is the safe default, but it can be *needlessly* conservative. - Because every sub-request paginates, splitting a large result further costs - little or no extra quota *as long as each sub-request still spans many + Because every chunk paginates, splitting a large result further costs + little or no extra quota *as long as each chunk still spans many pages* — rows-per-chunk far exceeding the page size (ten states pulled as one request page nearly as many times as ten per-state requests would). - When a split leaves each sub-request only a page or two, its partial final + When a split leaves each chunk only a page or two, its partial final page is extra, so finer chunks do add some requests. This context manager lets a caller who *knows* their pull is large ask for that finer split. The - trade is roughly the same pages for more, smaller sub-requests, which gives + trade is roughly the same pages for more, smaller chunks, which gives smoother progress, more even concurrency, and a smaller unit of retry/resume. @@ -118,21 +118,21 @@ def parallel_chunks(n: int) -> Iterator[None]: Parameters ---------- n : int - The number of sub-requests to fan the whole call out into — a positive + The number of chunks to fan the whole call out into — a positive integer such as ``2``, ``8``, or ``32``. It caps the plan's *total* - sub-request count (the cartesian product across every multi-value + chunk count (the cartesian product across every multi-value argument combined, not per argument), so several multi-value arguments cannot multiply past it. The cap is a ceiling, never exceeded: the actual count is bounded below by what the ~8 KB URL limit already forces and above by ``n``. So an ``n`` larger than the input allows - simply yields one sub-request per value, and with several multi-value + simply yields one chunk per value, and with several multi-value arguments the total may land somewhat below ``n`` because splits are whole (the plan can't always divide evenly onto ``n``). ``n=1`` asks for no extra fan-out. - Each sub-request fetches at least one page, so it costs at least one + Each chunk fetches at least one page, so it costs at least one request against your hourly rate limit — a larger ``n`` spends more - quota. How many sub-requests run *at once* is capped separately by + quota. How many chunks run *at once* is capped separately by ``API_USGS_CONCURRENT`` (default 32), so an ``n`` beyond that adds quota without adding parallelism; the useful range is roughly ``2`` up to ``API_USGS_CONCURRENT``. @@ -154,20 +154,20 @@ def parallel_chunks(n: int) -> Iterator[None]: getters already do for oversized requests; opting in just brings them to a request that would otherwise be a single call: - - ``max_rows``: each sub-request paginates up to ``max_rows`` rows + - ``max_rows``: each chunk paginates up to ``max_rows`` rows independently, then the combined result is sorted and truncated to ``max_rows``. So a call with ``max_rows`` set returns a *different* (though still valid and deterministically sorted) row set inside a ``parallel_chunks`` block than without one. The cap is drawn from the - union of the sub-requests, not a single stream. Don't pair a tight + union of the chunks, not a single stream. Don't pair a tight ``max_rows`` preview with ``parallel_chunks`` if you need exactly the rows the un-fanned call would return. - Resumability: a single request either fully succeeds or fully fails, but a fanned-out call can fail partway (e.g. a mid-call rate-limit) and raise a resumable :class:`~dataretrieval.ogc.interruptions.ChunkInterrupted` - (or ``QuotaExhausted``) carrying the completed sub-requests. Finish the + (or ``QuotaExhausted``) carrying the completed chunks. Finish the call with ``exc.call.resume()``. - - Cross-sub-request de-duplication keys on the feature ``id``; features + - Cross-chunk de-duplication keys on the feature ``id``; features with no ``id`` can't be deduped, so overlapping filter clauses split across chunks may yield duplicate rows. @@ -202,7 +202,7 @@ def multi_value_chunked( constructs a :class:`ChunkedCall` over the decorated ``async def fetch(args) -> (df, response)``, and drives it to completion via :meth:`ChunkedCall.resume`. The plan splits multi-value - list params and the cql-text filter so each sub-request URL fits the + list params and the cql-text filter so each chunk URL fits the byte limit. An already-fitting request is a one-step plan, unless an active :func:`parallel_chunks` block asks the plan to fan out more finely. See the module docstring for the concurrency model. @@ -238,7 +238,7 @@ def multi_value_chunked( See Also -------- ChunkPlan : Planning shape (axes, partitioning, passthrough). - ChunkedCall : Per-sub-request execution and resume semantics. + ChunkedCall : Per-chunk execution and resume semantics. """ def decorator( @@ -253,9 +253,9 @@ def wrapper( limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit # Read the parallel_chunks dial ``n`` from the ambient set by # ``parallel_chunks`` (1 = off outside any such block; otherwise the - # requested total sub-request cap). It only affects *planning*, done + # requested total chunk cap). It only affects *planning*, done # here up front, so a later resume — which re-issues the - # already-planned sub-requests — needs no snapshot. + # already-planned chunks — needs no snapshot. plan = ChunkPlan( args, build_request, limit, max_chunks=_parallel_chunks.get() ) @@ -271,7 +271,7 @@ def wrapper( canonical_url=plan.canonical_url, # The collection name, for the progress line the executor # opens. ``get_ogc_data`` puts it in ``args``. - service=args.get("service"), + service=args.get("collection"), ).resume() return wrapper diff --git a/dataretrieval/ogc/context.py b/dataretrieval/ogc/context.py index f416fd2a..ef9673d6 100644 --- a/dataretrieval/ogc/context.py +++ b/dataretrieval/ogc/context.py @@ -15,11 +15,15 @@ # OGC base URL targeted by request construction and schema lookup. Empty by # default *on purpose*: this package is API-neutral, so the adapter naming the -# service is the one that sets it (``get_ogc_data(base_url=...)`` does, and a -# hand-built request path such as ``waterdata.get_cql`` enters this context -# itself). A default endpoint here would silently send a caller that forgot to -# set it -- e.g. an NGWMN path -- to whichever API happened to be the default; -# an unset value instead fails loudly on the malformed URL. +# collection is the one that sets it (``get_ogc_data`` requires ``base_url`` and +# scopes it here, and a hand-built request path such as ``waterdata.get_cql`` +# enters this context itself). A default endpoint here would silently send a +# caller that forgot to set it -- e.g. an NGWMN path -- to whichever API +# happened to be the default, which is the worse failure. +# +# The empty default is not itself a good error -- see ``get_ogc_data``'s +# ``base_url`` docs for what it produces. The guard is that required argument, +# not this value. _ogc_base_url: Ambient[str] = Ambient("ogc_base_url", "") # Per-call request and response dialect. diff --git a/dataretrieval/ogc/engine.py b/dataretrieval/ogc/engine.py index d9f5a0b0..6b9e4a3c 100644 --- a/dataretrieval/ogc/engine.py +++ b/dataretrieval/ogc/engine.py @@ -15,10 +15,10 @@ API-specific behavior is supplied by the caller: * ``output_id`` — the user-facing column the wire ``id`` is renamed to, - passed explicitly (no service map lives here). + passed explicitly (no collection map lives here). * ``base_url`` — the OGC API base to target. * ``extra_id_cols`` — synthetic id columns to push to the end of a result. -* ``dialect`` — an :class:`OgcDialect` describing which services need +* ``dialect`` — an :class:`OgcDialect` describing which collections need POST/CQL2 and which use date-only (vs. full datetime) time arguments. """ @@ -131,7 +131,7 @@ async def _paginate( client: httpx.AsyncClient | None = None, raise_for_status: Callable[[httpx.Response], None] = _raise_for_non_200, ) -> tuple[pd.DataFrame, httpx.Response]: - """Compatibility wrapper around service-neutral cursor pagination.""" + """Compatibility wrapper around collection-neutral cursor pagination.""" session = client if client is not None else active_client() return await paginate( initial_req, @@ -216,11 +216,11 @@ async def follow_up(cursor: str, sess: httpx.AsyncClient) -> httpx.Response: def get_ogc_data( args: dict[str, Any], - service: str, + collection: str, output_id: str, *, + base_url: str, max_rows: int | None = None, - base_url: str | None = None, extra_id_cols: frozenset[str] | set[str] = frozenset(), dialect: OgcDialect | None = None, ) -> tuple[pd.DataFrame, BaseMetadata]: @@ -234,29 +234,33 @@ def get_ogc_data( Parameters ---------- args : Dict[str, Any] - Dictionary of request arguments for the OGC service. - service : str + Dictionary of request arguments for the OGC collection. + collection : str The OGC API collection name (e.g., ``"daily"``, ``"monitoring-locations"``, ``"continuous"``). output_id : str The user-facing id column the wire ``id`` is renamed to. Required — - the per-API service-to-id map lives in the caller, not here. + the per-API collection-to-id map lives in the caller, not here. max_rows : int, optional Stop paginating once this many rows have been collected and truncate the result to exactly ``max_rows``. ``None`` (default) fetches the full result. Intended for cheap previews of large, un-chunked tables (e.g. :func:`get_reference_table`). - base_url : str, optional - OGC API base URL to target. Required in practice -- this package is - API-neutral and names no service of its own; each adapter passes its - own base (e.g. ``waterdata.utils.OGC_API_URL``, - ``ngwmn.NGWMN_OGC_API_URL``). Falls back to the base URL already in - scope for the current call. + base_url : str + OGC API base URL to target. Required: this package is API-neutral and + names no API of its own, so each adapter passes its own base (e.g. + ``waterdata.utils.OGC_API_URL``, ``ngwmn.NGWMN_OGC_API_URL``). It was + once optional, falling back to whatever was in ambient scope -- which + defaults to the empty string, so omitting it built a *relative* + ``/collections/{id}/items`` that planning accepted and only httpx + rejected at send time, surfacing as a NetworkError about an unknown + service. Requiring it moves that mistake to the call site, where mypy + catches it. extra_id_cols : set or frozenset, optional Synthetic id columns to push to the end of a result frame (see :func:`_arrange_cols`). Defaults to an empty set. dialect : OgcDialect, optional - Per-API request quirks (CQL2-only services, date-only services). + Per-API request quirks (CQL2-only collections, date-only collections). Defaults to a plain OGC API with neither. Returns @@ -270,7 +274,7 @@ def get_ogc_data( ----- - The function does not mutate the input `args` dictionary. - Handles optional arguments such as `convert_type`. - - Applies column cleanup and reordering based on service and properties. + - Applies column cleanup and reordering based on collection and properties. """ # Enforce a genuine positive integer up front: a float (even ``10.0``) or # ``bool`` would pass a bare ``< 1`` check and then crash deep in @@ -281,17 +285,14 @@ def get_ogc_data( if dialect is None: dialect = _DEFAULT_DIALECT - if base_url is None: - base_url = _ogc_base_url.get() - args = args.copy() - args["service"] = service - args = _switch_arg_id(args, id_name=output_id, service=service) + args["collection"] = collection + args = _switch_arg_id(args, id_name=output_id, collection=collection) # Capture `properties` before the id-switch so post-processing sees # the user-facing names, not the wire-format ones. properties = args.get("properties") args["properties"] = _switch_properties_id( - properties, id_name=output_id, service=service + properties, id_name=output_id, collection=collection ) convert_type = args.pop("convert_type", False) args = {k: v for k, v in args.items() if v is not None} @@ -302,13 +303,13 @@ def get_ogc_data( # this function). ``_finalize_ogc`` is the single source of result shape; # it also applies ``max_rows`` to the *combined* frame so the cap is the # exact total even when the plan chunks or the call is resumed, while - # ``_row_cap`` below only early-stops each sub-request's pagination. + # ``_row_cap`` below only early-stops each chunk's pagination. finalize = functools.partial( _finalize_ogc, properties=properties, output_id=output_id, convert_type=convert_type, - service=service, + collection=collection, max_rows=max_rows, extra_id_cols=extra_id_cols, dialect=dialect, @@ -328,10 +329,10 @@ async def _fetch_once( ``@chunking.multi_value_chunked`` models every multi-value list parameter and the cql-text filter as a chunkable axis, greedy-halves - the biggest chunk across all axes until each sub-request URL fits, + the biggest chunk across all axes until each chunk URL fits, and iterates the cartesian product. With no chunkable inputs the decorator passes args through unchanged. The decorator gathers every - sub-request over one shared :class:`httpx.AsyncClient` (concurrency + chunk over one shared :class:`httpx.AsyncClient` (concurrency bounded by a semaphore, sized from ``API_USGS_CONCURRENT``). It also returns a *synchronous* wrapper, so ``get_ogc_data`` keeps calling ``_fetch_once(args, finalize=...)`` synchronously. The return shape is @@ -344,7 +345,7 @@ async def _fetch_once( def fetch_ogc_request( request: httpx.Request, *, - service: str, + collection: str, ) -> tuple[pd.DataFrame, httpx.Response]: """Execute a prepared OGC request with pagination, returning (df, response). @@ -360,7 +361,7 @@ def fetch_ogc_request( ---------- request : httpx.Request A fully-constructed OGC API request (typically a POST/CQL2). - service : str + collection : str Collection name, used only for progress-context labelling. Returns @@ -379,5 +380,5 @@ async def _fetch(req: httpx.Request) -> tuple[pd.DataFrame, httpx.Response]: _fetch, RetryPolicy.from_env(), canonical_url=str(request.url), - service=service, + service=collection, ).resume() diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index d6e07ff7..e5af5411 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -2,7 +2,7 @@ This module holds the side-effect-free planning half of the chunker: deciding how to split one over-budget OGC request into URL-fitting -sub-requests (:class:`ChunkPlan` and the axis/byte-accounting helpers). +chunks (:class:`ChunkPlan` and the axis/byte-accounting helpers). It has no event loop, retry policy, or network state — those live in :mod:`dataretrieval.ogc.chunking` (resumable execution) and :mod:`dataretrieval.transport.retry` (retry policy), which import the plan and @@ -98,7 +98,7 @@ def _safe_request_bytes( url_limit: int, ) -> int: """ - Size a candidate sub-request, treating ``httpx.InvalidURL`` as "too large". + Size a candidate chunk, treating ``httpx.InvalidURL`` as "too large". ``httpx.URL`` enforces a hard 64 KB cap per URL component (``MAX_URL_LENGTH``) and raises ``httpx.InvalidURL`` for anything @@ -111,7 +111,7 @@ def _safe_request_bytes( build_request : Callable[..., httpx.Request] Factory that turns a kwargs dict into a sized request. args : dict[str, Any] - Per-sub-request kwargs to pass through to ``build_request``. + Per-chunk kwargs to pass through to ``build_request``. url_limit : int The chunker's byte budget; returned + 1 on overflow. @@ -145,7 +145,7 @@ class _Axis: ---------- arg_key : str The args-dict key this axis substitutes back into when a - sub-request is rendered. + chunk is rendered. atoms : tuple of str The smallest indivisible units along this axis (one site, one OR-clause, …). A "chunk" is a contiguous slice of ``atoms``. @@ -246,7 +246,7 @@ def _split_at(chunks: list[list[str]], idx: int) -> None: The single primitive both planning passes use to fan an axis out. It preserves the partition invariants every consumer relies on: *coverage* (each atom survives, exactly once) and *contiguous, deterministic order* - (resume and :meth:`ChunkPlan.iter_sub_args` depend on it). Kept in one + (resume and :meth:`ChunkPlan.iter_chunk_args` depend on it). Kept in one place so those invariants can't drift between :meth:`ChunkPlan._plan` (byte-driven) and :meth:`ChunkPlan._refine` (fan-out-driven). """ @@ -257,16 +257,16 @@ def _split_at(chunks: list[list[str]], idx: int) -> None: class ChunkPlan: """ - Strategy for issuing one user-level request as URL-fitting sub-requests. + Strategy for issuing one user-level request as URL-fitting chunks. - Every sub-request URL fits ``url_limit``. Constructing a plan *is* planning: + Every chunk URL fits ``url_limit``. Constructing a plan *is* planning: ``ChunkPlan(args, build_request, url_limit)`` extracts the chunkable axes, runs greedy halving on the biggest chunk across all axes, and stores the result. Passthrough requests (no chunkable axes, or already fitting) are represented as a trivial plan with empty ``axes`` / ``chunks`` and - ``total == 1``; :meth:`iter_sub_args` yields the original args + ``total == 1``; :meth:`iter_chunk_args` yields the original args unchanged so the ``ChunkedCall`` loop is the same shape either way. @@ -279,18 +279,18 @@ class ChunkPlan: e.g. ``_construct_api_requests``. url_limit : int Byte budget for the request (URL + body) — a hard ceiling every - sub-request must fit. + chunk must fit. max_chunks : int, optional - Hard cap on the plan's total sub-request count (default ``1`` = off). + Hard cap on the plan's total chunk count (default ``1`` = off). ``1`` chunks only as much as ``url_limit`` requires — the most - conservative plan, fewest sub-requests — so a fitting request is a + conservative plan, fewest chunks — so a fitting request is a passthrough. A cap of ``2`` or more fans the plan out to up to - ``max_chunks`` sub-requests overall (the cartesian product across axes, + ``max_chunks`` chunks overall (the cartesian product across axes, never fewer than the byte budget already forces). The cap applies to the plan as a whole, not per axis, so several multi-value axes can't multiply past it. The plan never exceeds the cap and may land below it when no whole split lands on it exactly. ``max_chunks`` is a - sub-request count, so a value below ``1`` (``0`` or negative) is a + chunk count, so a value below ``1`` (``0`` or negative) is a caller error and raises ``ValueError``. Set from the :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see :meth:`_refine`. @@ -299,7 +299,7 @@ class ChunkPlan: ---------- args : dict The original user-level args this plan was built for. Bound to - the plan so :meth:`iter_sub_args` is self-contained. + the plan so :meth:`iter_chunk_args` is self-contained. axes : list[_Axis] The chunkable axes of ``args``: each multi-value list parameter, plus the cql-text filter (if any) split on top-level @@ -330,7 +330,7 @@ def __init__( max_chunks: int = 1, ) -> None: if max_chunks < 1: - # ``max_chunks`` is a sub-request *count*: the minimum is ``1`` + # ``max_chunks`` is a chunk *count*: the minimum is ``1`` # (the ambient default outside any ``parallel_chunks`` block), # which means "off — no extra fan-out". ``0`` or negative is a # meaningless count and can only be a caller bug, so fail loudly @@ -376,7 +376,7 @@ def __init__( # "needs chunking" signal, so swallow it and proceed to plan. # When the unchunked URL does build, preserve it as ``canonical_url`` # so ``BaseMetadata.url`` echoes the user's original query verbatim. - # Only fall back to a worst-case sub-request URL when the URL itself + # Only fall back to a worst-case chunk URL when the URL itself # can't be constructed. try: initial_request = build_request(**args) @@ -390,7 +390,7 @@ def __init__( # A request that already fits and hasn't opted into finer chunking is # the common passthrough: leave ``axes``/``chunks`` empty so - # ``total == 1`` and ``iter_sub_args`` yields the original args + # ``total == 1`` and ``iter_chunk_args`` yields the original args # verbatim. ``max_chunks == 1`` (off / no extra fan-out) means # "don't split", so it takes this path; only ``max_chunks >= 2`` asks # for extra fan-out and sets the axes up to be refined below. @@ -400,7 +400,7 @@ def __init__( self.axes = axes self.chunks = {axis.arg_key: [list(axis.atoms)] for axis in axes} if not fits: - # Hard pass: greedy-halve until every worst-case sub-request fits + # Hard pass: greedy-halve until every worst-case chunk fits # the byte budget (may raise ``Unchunkable``). self._plan(build_request, url_limit) # Soft pass: optionally split further than the byte budget requires. @@ -410,7 +410,7 @@ def __init__( if self.canonical_url is None: # Original URL was un-constructable (httpx.InvalidURL); fall - # back to the worst-case sub-request URL so + # back to the worst-case chunk URL so # ``BaseMetadata.url`` still surfaces something # informative. If even that overflows, leave canonical_url # as None (set above) and let the response's own URL stand. @@ -425,7 +425,7 @@ def _plan( """ Greedy-halve the biggest chunk across axes until every URL fits. - Halving continues until the worst-case sub-request URL fits + Halving continues until the worst-case chunk URL fits ``url_limit``, mutating ``self.chunks`` in place. List axes and the filter axis are treated uniformly — each is just a list of atoms joined by its axis's separator. @@ -456,7 +456,7 @@ def _plan( raise Unchunkable( f"Request exceeds {url_limit} bytes (URL + body) at the " f"smallest reducible plan (every axis at one atom per " - f"sub-request). Reduce input sizes, shorten or simplify " + f"chunk). Reduce input sizes, shorten or simplify " f"the filter, or split the call manually." ) _split_at(self.chunks[biggest_axis.arg_key], biggest_idx) @@ -472,7 +472,7 @@ def _refine(self, max_chunks: int) -> None: below the cap). Implementation. Each split multiplies the plan by ``(k+1)/k`` for the - chosen axis (adding ``total // k`` sub-requests, not one), so a split + chosen axis (adding ``total // k`` chunks, not one), so a split is taken only when it keeps :attr:`total` within the cap. When no in-budget split remains, the plan stops *below* the cap rather than overshooting (two even axes can reach 4 but not 5, so a cap of 5 yields @@ -498,12 +498,12 @@ def _refine(self, max_chunks: int) -> None: return # Largest splittable chunk among the axes whose split still fits the # cap. Splitting any chunk of an axis with ``k`` chunks turns that - # ``k`` into ``k+1``, so it adds ``total // k`` sub-requests (the + # ``k`` into ``k+1``, so it adds ``total // k`` chunks (the # product of the other axes) regardless of which chunk. Hence the # budget test is per axis, not per chunk. Skipping an over-budget # axis makes ``max_chunks`` a true ceiling. The ranking key is atom # count (``len``), not URL bytes like ``_plan`` — this pass balances - # work across sub-requests rather than fitting a byte budget. A + # work across chunks rather than fitting a byte budget. A # chunk of size 1 can't be split further. Stable input order breaks # ties by axis order, then lowest index within an axis. candidate: tuple[_Axis, int] | None = None @@ -526,7 +526,7 @@ def _refine(self, max_chunks: int) -> None: def _worst_case_args(self) -> dict[str, Any]: """ - Args for the largest sub-request the current partition will issue. + Args for the largest chunk the current partition will issue. Each axis contributes its longest chunk (by URL-encoded bytes), rendered back into the args dict. @@ -540,7 +540,7 @@ def _worst_case_args(self) -> dict[str, Any]: @property def total(self) -> int: """ - Total sub-request count: product of per-axis chunk counts. + Total chunk count: product of per-axis chunk counts. Returns ------- @@ -550,9 +550,9 @@ def total(self) -> int: """ return math.prod((len(self.chunks[ax.arg_key]) for ax in self.axes), start=1) - def iter_sub_args(self) -> Iterator[dict[str, Any]]: + def iter_chunk_args(self) -> Iterator[dict[str, Any]]: """ - Yield substituted args for each sub-request, in deterministic order. + Yield substituted args for each chunk, in deterministic order. The order is the cartesian product over axes in extraction order. The same plan yields the same sub-args sequence on every invocation, so @@ -569,12 +569,12 @@ def iter_sub_args(self) -> Iterator[dict[str, Any]]: return chunk_lists = [self.chunks[ax.arg_key] for ax in self.axes] for combo in itertools.product(*chunk_lists): - sub_args = dict(self.args) + chunk_args = dict(self.args) for axis, chunk in zip(self.axes, combo, strict=False): - sub_args[axis.arg_key] = axis.render(chunk) - yield sub_args + chunk_args[axis.arg_key] = axis.render(chunk) + yield chunk_args - # ``total`` and ``iter_sub_args`` are this class's domain vocabulary and + # ``total`` and ``iter_chunk_args`` are this class's domain vocabulary and # stay as they are. The dunders are how a plan satisfies # :class:`~dataretrieval.transport.fanout.FanOutPlan`, which asks for a # sized iterable and nothing chunking-specific. They delegate rather than @@ -583,4 +583,4 @@ def __len__(self) -> int: return self.total def __iter__(self) -> Iterator[dict[str, Any]]: - return self.iter_sub_args() + return self.iter_chunk_args() diff --git a/dataretrieval/ogc/policy.py b/dataretrieval/ogc/policy.py index 2f9f58c5..99498ec1 100644 --- a/dataretrieval/ogc/policy.py +++ b/dataretrieval/ogc/policy.py @@ -5,12 +5,12 @@ It depends only on the stdlib, so any OGC submodule can import it without creating cycles. -It names no endpoint: which service an OGC call targets is the *adapter's* +It names no endpoint: which collection an OGC call targets is the *adapter's* policy, supplied per call as ``base_url`` (see :data:`dataretrieval.ogc.context._ogc_base_url`). A default here would quietly point every generic OGC caller at one API. -It must NOT import engine, shaping, or any service adapter. +It must NOT import engine, shaping, or any collection adapter. """ from __future__ import annotations diff --git a/dataretrieval/ogc/requests.py b/dataretrieval/ogc/requests.py index 882a8b5f..fa3de029 100644 --- a/dataretrieval/ogc/requests.py +++ b/dataretrieval/ogc/requests.py @@ -37,29 +37,29 @@ # --------------------------------------------------------------------------- -def _switch_arg_id(ls: dict[str, Any], id_name: str, service: str) -> dict[str, Any]: +def _switch_arg_id(ls: dict[str, Any], id_name: str, collection: str) -> dict[str, Any]: """Switch argument id from its package-specific identifier to the standardized "id" key that the API recognizes.""" - service_id = service.replace("-", "_") + "_id" + collection_id = collection.replace("-", "_") + "_id" if "id" not in ls: - if service_id in ls: - ls["id"] = ls[service_id] + if collection_id in ls: + ls["id"] = ls[collection_id] elif id_name in ls: ls["id"] = ls[id_name] - ls.pop(service_id, None) + ls.pop(collection_id, None) ls.pop(id_name, None) return ls def _switch_properties_id( - properties: list[str] | None, id_name: str, service: str + properties: list[str] | None, id_name: str, collection: str ) -> list[str]: """Build the wire ``properties`` list, dropping every id alias and ``geometry``.""" if not properties: return [] - service_id = service.replace("-", "_") + "_id" - drop = {"id", "geometry", id_name, service_id} + collection_id = collection.replace("-", "_") + "_id" + drop = {"id", "geometry", id_name, collection_id} normalized = (p.replace("-", "_") for p in properties) return [p for p in normalized if p not in drop] @@ -120,9 +120,9 @@ def _partition_request_params( return get_params, {} -def _items_url(service: str) -> str: - """The OGC items endpoint for ``service`` under the active base URL.""" - return f"{_ogc_base_url.get()}/collections/{service}/items" +def _items_url(collection: str) -> str: + """The OGC items endpoint for ``collection`` under the active base URL.""" + return f"{_ogc_base_url.get()}/collections/{collection}/items" def _cql2_post_request( @@ -141,24 +141,26 @@ def _cql2_post_request( def _construct_api_requests( - service: str, + collection: str, properties: list[str] | None = None, bbox: list[float] | None = None, limit: int | None = None, skip_geometry: bool | None = None, **kwargs: Any, ) -> httpx.Request: - """Construct an HTTP request object for the specified OGC API service.""" - service_url = _items_url(service) + """Construct an HTTP request object for the specified OGC API collection.""" + service_url = _items_url(collection) dialect = _dialect.get() for key in _DATE_RANGE_PARAMS: if key in kwargs: kwargs[key] = _format_api_dates( kwargs[key], - date=service in dialect.date_only_services and key != "last_modified", + date=( + collection in dialect.date_only_services and key != "last_modified" + ), ) params, post_params = _partition_request_params( - kwargs, use_cql2=service in dialect.cql2_services + kwargs, use_cql2=collection in dialect.cql2_services ) _ogc_query_params( @@ -185,7 +187,7 @@ def _construct_api_requests( def _construct_cql_request( - service: str, + collection: str, cql_body: str, *, properties: list[str] | None = None, @@ -194,7 +196,7 @@ def _construct_cql_request( skip_geometry: bool | None = None, ) -> httpx.Request: """Build a POST/CQL2 request from a verbatim CQL2 body.""" - service_url = _items_url(service) + service_url = _items_url(collection) params = _ogc_query_params( {}, properties=properties, @@ -294,7 +296,10 @@ def prepare_request_args( by forgetting to union them back in. """ no_normalize = _NO_NORMALIZE_PARAMS | frozenset(extra_no_normalize) - to_exclude = {"service", "output_id"} + # Both spellings: this drops the caller's collection selector out of the + # query string, and public getters still name that local ``service`` + # (waterdata.get_samples) or ``service=`` during the get_cql deprecation. + to_exclude = {"collection", "service", "output_id"} if exclude: to_exclude.update(exclude) diff --git a/dataretrieval/ogc/schema.py b/dataretrieval/ogc/schema.py index c3b3fd23..25df084f 100644 --- a/dataretrieval/ogc/schema.py +++ b/dataretrieval/ogc/schema.py @@ -1,4 +1,4 @@ -"""Asking an OGC service to describe itself. +"""Asking an OGC collection to describe itself. Queryables and collection schemas: which properties a collection accepts, and what columns it returns. Separate from request construction because answering @@ -13,7 +13,6 @@ import pandas as pd from dataretrieval._response_metadata import BaseMetadata -from dataretrieval.ogc.context import _ogc_base_url from dataretrieval.ogc.errors import _raise_for_non_200 from dataretrieval.transport.http import HTTPX_DEFAULTS from dataretrieval.transport.http import default_headers as _default_headers @@ -21,17 +20,15 @@ def _check_ogc_requests( - endpoint: str, req_type: str = "queryables", *, base_url: str | None = None + endpoint: str, req_type: str = "queryables", *, base_url: str ) -> tuple[dict[str, Any], httpx.Response]: """Retrieve one collection's queryables or response schema. ``base_url`` names the API to ask; it defaults to the one in scope for the - current call rather than to any particular service. + current call rather than to any particular collection. """ if req_type not in ("queryables", "schema"): raise ValueError(f"req_type must be 'queryables' or 'schema', got {req_type!r}") - if base_url is None: - base_url = _ogc_base_url.get() url = f"{base_url}/collections/{endpoint}/{req_type}" response = _get(url, headers=_default_headers(url), **HTTPX_DEFAULTS) _raise_for_non_200(response) @@ -39,11 +36,11 @@ def _check_ogc_requests( def queryables_frame( - collection: str, *, base_url: str | None = None + collection: str, *, base_url: str ) -> tuple[pd.DataFrame, BaseMetadata]: """Tabulate one collection's queryable properties. - Reading an OGC queryables document is protocol knowledge, not service + Reading an OGC queryables document is protocol knowledge, not collection knowledge, so it lives here rather than in any one API's getters -- every OGC adapter in the package can offer the same table. ``base_url`` names the API to ask, defaulting to the one in scope for the current call. diff --git a/dataretrieval/ogc/shaping.py b/dataretrieval/ogc/shaping.py index 2cb1befc..e9d26b32 100644 --- a/dataretrieval/ogc/shaping.py +++ b/dataretrieval/ogc/shaping.py @@ -120,7 +120,7 @@ def _get_resp_data( The non-geopandas branch normalizes each feature's ``properties`` object, flattening nested dictionaries with an underscore separator, then adds the top-level ``id`` and a ``geometry`` column containing the coordinates. The - ``id`` column is always added so the downstream service-specific rename + ``id`` column is always added so the downstream collection-specific rename works even when all IDs are missing; ``geometry`` is added only when coordinates are present. Feature-level envelope fields are deliberately excluded. @@ -143,7 +143,7 @@ def _get_resp_data( properties = [feature.get("properties") or {} for feature in features] df = pd.json_normalize(properties, sep="_") # Always materialize the feature-level ID (possibly all-None) so - # ``_arrange_cols`` can perform the documented service-specific rename. + # ``_arrange_cols`` can perform the documented collection-specific rename. df["id"] = [feature.get("id") for feature in features] _attach_coordinates(df, features) return df @@ -169,7 +169,7 @@ def _get_resp_data( def _deal_with_empty( return_list: pd.DataFrame, properties: list[str] | None, - service: str, + collection: str, *, base_url: str | None = None, ) -> pd.DataFrame: @@ -178,7 +178,7 @@ def _deal_with_empty( If `return_list` is empty, determines the column names to use: - If `properties` is not provided or contains only NaN values, - retrieves schema properties from the specified service. + retrieves schema properties from the specified collection. - Otherwise, uses the provided `properties` list as column names. Parameters @@ -187,11 +187,11 @@ def _deal_with_empty( The DataFrame to check for emptiness. properties : Optional[List[str]] List of property names to use as columns, or None. - service : str - The service endpoint to query for schema properties if needed. + collection : str + The collection endpoint to query for schema properties if needed. base_url : str, optional OGC API base URL to use for that schema query. Defaults to the base - URL in scope for the current call, not to any particular service. + URL in scope for the current call, not to any particular collection. Returns ------- @@ -207,7 +207,7 @@ def _deal_with_empty( from dataretrieval.ogc.schema import _check_ogc_requests schema, _ = _check_ogc_requests( - endpoint=service, req_type="schema", base_url=base_url + endpoint=collection, req_type="schema", base_url=base_url ) properties = list(schema.get("properties", {}).keys()) return pd.DataFrame(columns=properties) @@ -252,8 +252,8 @@ def _arrange_cols( local_properties = list(properties) if "geometry" in df.columns and "geometry" not in local_properties: local_properties.append("geometry") - # 'id' is a valid service column, but expose it under the - # service-specific output_id name instead. + # 'id' is a valid collection column, but expose it under the + # collection-specific output_id name instead. if "id" in local_properties: local_properties[local_properties.index("id")] = output_id df = df.loc[:, [col for col in local_properties if col in df.columns]] @@ -363,7 +363,7 @@ def _finalize_ogc( properties: list[str] | None, output_id: str, convert_type: bool, - service: str, + collection: str, max_rows: int | None = None, extra_id_cols: frozenset[str] | set[str] = frozenset(), dialect: OgcDialect | None = None, @@ -384,7 +384,7 @@ def _finalize_ogc( the chunker's raw frame and bare ``httpx.Response``. ``max_rows`` is applied here (after dedup/sort, on the *combined* frame) - rather than only per-sub-request, so a chunked call's total is bounded + rather than only per-chunk, so a chunked call's total is bounded to exactly ``max_rows`` and a resumed call honors the cap too. The per-``_paginate`` ``_row_cap`` is only an early-stop download bound. ``base_url`` is captured with the finalizer so resumed calls query the same @@ -394,7 +394,7 @@ def _finalize_ogc( dialect = DEFAULT_DIALECT if base_url is None: base_url = _ogc_base_url.get() - frame = _deal_with_empty(frame, properties, service, base_url=base_url) + frame = _deal_with_empty(frame, properties, collection, base_url=base_url) # Normalize to PEP-8 snake_case column names *first*, so the dialect's # ``time_cols``/``numerical_cols``/``sort_cols`` (all snake_case) match # regardless of whether the API returns snake_case (Water Data, where diff --git a/dataretrieval/progress.py b/dataretrieval/progress.py index 9f51cf76..d59bff48 100644 --- a/dataretrieval/progress.py +++ b/dataretrieval/progress.py @@ -123,7 +123,7 @@ def __init__( # The hourly request quota (``x-ratelimit-limit``), shown as the # denominator when the server reports it. self.rate_limit: str | None = None - # Transient note shown while a sub-request backs off before a + # Transient note shown while a chunk backs off before a # retry; cleared by the next page/chunk so it doesn't linger. self.retry_note: str | None = None self._last_len = 0 @@ -157,7 +157,7 @@ def add_page(self, rows: int = 0) -> None: self._render() def note_retry(self, *, attempt: int, wait: float) -> None: - """Show that a sub-request is backing off before retry ``attempt``. + """Show that a chunk is backing off before retry ``attempt``. Cleared by the next :meth:`add_page` / :meth:`start_chunk` (or by :meth:`close`) so the line returns to normal once the retry resolves. diff --git a/dataretrieval/transport/fanout.py b/dataretrieval/transport/fanout.py index 80b17c72..33d3859d 100644 --- a/dataretrieval/transport/fanout.py +++ b/dataretrieval/transport/fanout.py @@ -1,4 +1,4 @@ -"""Bounded, resumable fan-out execution over a plan of sub-requests. +"""Bounded, resumable fan-out execution over a plan of chunks. A fan-out is one logical query the service forces into several requests. Two unrelated reasons produce one: @@ -22,24 +22,24 @@ supplies a :class:`FanOutPlan` (whatever structure it divided into, if any) and an ``async def fetch(item) -> (df, response)``. -Concurrency: :meth:`FanOut._run` dispatches every pending sub-request under one +Concurrency: :meth:`FanOut._run` dispatches every pending chunk under one ``asyncio.gather`` sharing a single ``httpx.AsyncClient``. An ``asyncio.Semaphore`` -- not the client's connection pool, which is merely sized -to match -- caps the sub-requests in flight at ``N``; see :meth:`FanOut._run` +to match -- caps the chunks in flight at ``N``; see :meth:`FanOut._run` for why the gate must be the semaphore rather than the pool. -``API_USGS_CONCURRENT`` resolves ``N``: an integer N > 1 allows N sub-requests +``API_USGS_CONCURRENT`` resolves ``N``: an integer N > 1 allows N chunks in flight; ``1`` forces sequential dispatch; the literal ``unbounded`` lifts the -cap. ``N`` bounds only how many of a query's sub-requests are in flight at once +cap. ``N`` bounds only how many of a query's chunks are in flight at once -- a client-side trade-off between open connections and fan-out latency. It does not affect the API rate limit: a fanned-out call issues the same number of -sub-requests regardless of ``N``, so ``N`` changes their timing, not the total +chunks regardless of ``N``, so ``N`` changes their timing, not the total request volume. The USGS API rate-limits by volume over time (HTTP 429), not by simultaneity; set ``API_USGS_PAT`` to raise that quota. The default of 32 is a conservative cap that keeps connection use modest. The fan-out runs in a short-lived worker thread (an ``anyio`` blocking portal), so it works whether or not the caller is already inside an event loop (Jupyter / IPython / async apps). -Retries: each sub-request is retried on a transient failure (429, 5xx, +Retries: each chunk is retried on a transient failure (429, 5xx, connect/read timeout) with exponential backoff + full jitter, honoring a server ``Retry-After`` when present. ``API_USGS_RETRIES`` sets the cap (default 4; ``0`` disables). A ``Retry-After`` longer than the per-call ceiling escalates to @@ -47,9 +47,9 @@ Interruption: any mid-stream transient failure surfaces as a :class:`~dataretrieval.interruptions.FanOutInterrupted` subclass carrying -``.call``, a :class:`FanOut` handle owning the already-completed sub-request +``.call``, a :class:`FanOut` handle owning the already-completed chunk state. Call ``.call.resume()`` once the underlying condition clears; only the -still-pending sub-requests are re-issued. +still-pending chunks are re-issued. """ from __future__ import annotations @@ -81,13 +81,13 @@ from dataretrieval.transport.retry import _NO_RETRY, RetryPolicy from dataretrieval.transport.retry import retry_async as _retry -#: One sub-request's description, as the adapter's ``fetch`` wants it. The +#: One chunk's description, as the adapter's ``fetch`` wants it. The #: executor never inspects it — see :class:`FanOutPlan`. -_Item = TypeVar("_Item") +_Chunk = TypeVar("_Chunk") #: The same thing in :class:`FanOutPlan`, where it only ever comes *out* of the #: plan. Covariant so a ``list[httpx.Request]`` satisfies a plan of any #: supertype, the way ``Iterable`` is covariant for the same reason. -_ItemCo = TypeVar("_ItemCo", covariant=True) +_ChunkCo = TypeVar("_ChunkCo", covariant=True) # Fan-out concurrency cap, read at call time (not import) so test # ``monkeypatch.setenv`` applies. Value grammar in :func:`_read_concurrency_env`; @@ -122,7 +122,7 @@ def _resolve_concurrency(default: int = _CONCURRENCY_DEFAULT) -> int | None: Returns ------- int or None - ``1`` for sequential dispatch (one sub-request at a time); an + ``1`` for sequential dispatch (one chunk at a time); an integer >1 for bounded concurrency; ``None`` to disable the per-call cap entirely (the ``unbounded`` keyword). """ @@ -146,19 +146,25 @@ def _resolve_concurrency(default: int = _CONCURRENCY_DEFAULT) -> int | None: # --------------------------------------------------------------------------- -class FanOutPlan(Protocol[_ItemCo]): +class FanOutPlan(Protocol[_ChunkCo]): """ - A fan-out's shape: how many sub-requests, and what each one is. + The contract a plan satisfies for a fan-out to execute it. - Deliberately the two standard protocols rather than bespoke members. A - plan is a sized, iterable collection of sub-request descriptions, which is - exactly ``__len__`` + ``__iter__`` — so a plain ``list`` of pre-built + A **plan** is defined in ``CONTEXT.md``. This protocol is that enumeration + and nothing more, which is why it is named for the role it plays here + rather than for its contents: + :class:`~dataretrieval.ogc.planning.ChunkPlan` is *a* plan, and so is a + plain list of requests. + + Deliberately the two standard protocols rather than bespoke members, since + an enumeration of chunks is exactly ``__len__`` + ``__iter__`` -- so a + plain ``list`` of pre-built requests satisfies this with no adapter class, and a real planner satisfies it by delegating (see :class:`~dataretrieval.ogc.planning.ChunkPlan`, whose domain vocabulary is - ``total`` / ``iter_sub_args``). Naming them ``total`` and - ``iter_sub_args`` here would mean two names for ``len`` that could report - different counts, and a shim class for every adapter whose sub-requests + ``total`` / ``iter_chunk_args``). Naming them ``total`` and + ``iter_chunk_args`` here would mean two names for ``len`` that could report + different counts, and a shim class for every adapter whose chunks are already a list. The item type is whatever an adapter's own ``fetch`` accepts: this executor @@ -168,7 +174,7 @@ class FanOutPlan(Protocol[_ItemCo]): Iteration order is load-bearing: :meth:`FanOut.resume` keys completed work by position, so a plan that yielded a different order on a second pass - would resume the wrong sub-requests. ``len`` must agree with the number of + would resume the wrong chunks. ``len`` must agree with the number of items iteration yields — the usual contract for a sized collection. The identity of the query as a whole is *not* here: it is a value stamped @@ -178,7 +184,7 @@ class FanOutPlan(Protocol[_ItemCo]): def __len__(self) -> int: ... - def __iter__(self) -> Iterator[_ItemCo]: ... + def __iter__(self) -> Iterator[_ChunkCo]: ... # --------------------------------------------------------------------------- @@ -187,7 +193,7 @@ def __iter__(self) -> Iterator[_ItemCo]: ... # The per-call ``httpx.AsyncClient``, published for the duration of # ``FanOut._run`` so paginated-loop helpers reuse the same connection pool -# across every sub-request. ``None`` outside a fan-out — paginated helpers then +# across every chunk. ``None`` outside a fan-out — paginated helpers then # open their own short-lived client. Deliberately a plain ContextVar-backed # ambient rather than a parameter: the fetch closure an adapter injects is often # several frames below the client's owner. @@ -213,10 +219,10 @@ def active_client() -> httpx.AsyncClient | None: # Type aliases for the FanOut contract # --------------------------------------------------------------------------- -# The per-sub-request fetcher an adapter injects and ``FanOut`` drives: an +# The per-chunk fetcher an adapter injects and ``FanOut`` drives: an # ``async def fetch(item) -> (df, response)``, where ``item`` is whatever the # adapter's plan yields. -_Fetch = Callable[[_Item], Awaitable[tuple[pd.DataFrame, httpx.Response]]] +_Fetch = Callable[[_Chunk], Awaitable[tuple[pd.DataFrame, httpx.Response]]] # Caller-supplied transform applied to the combined result, so a resumed call # returns the same shape as an un-interrupted one rather than the executor's raw @@ -233,17 +239,17 @@ def _passthrough_result( return frame, response -class FanOut(Generic[_Item]): +class FanOut(Generic[_Chunk]): """ Stateful handle for a fanned-out call. - Holds the in-flight state (per-sub-request frames and responses) + Holds the in-flight state (per-chunk frames and responses) and the async fetcher. A single :meth:`resume` entry point drives the call from wherever it is to completion — used both for the first invocation and for subsequent retries after a :class:`~dataretrieval.interruptions.FanOutInterrupted`. - :meth:`_run` gathers every pending sub-request over one shared + :meth:`_run` gathers every pending chunk over one shared :class:`httpx.AsyncClient`, applies the failure-precedence rules, and combines; :meth:`resume` drives it through an ``anyio`` blocking portal so it works whether or not the caller is already inside an @@ -266,11 +272,11 @@ class FanOut(Generic[_Item]): Parameters ---------- plan : FanOutPlan - The sub-requests to execute: anything sized and iterable, from a + The chunks to execute: anything sized and iterable, from a :class:`~dataretrieval.ogc.planning.ChunkPlan` to a plain ``list`` of pre-built requests. fetch : Callable - ``async def`` that issues a single sub-request, given one item from + ``async def`` that issues a single chunk, given one item from ``plan``, and returns ``(frame, response)``. client_options : dict, optional Extra ``httpx.AsyncClient`` options for the shared client this run @@ -281,7 +287,7 @@ class FanOut(Generic[_Item]): canonical_url : str or None, optional URL identifying the query as a whole, restored onto the combined response so the caller sees the request they made rather than - whichever sub-request happened to land last. Also the destination + whichever chunk happened to land last. Also the destination :meth:`resume` labels its progress line with. service : str or None, optional Human-facing name of what is being retrieved (e.g. ``"daily"``, @@ -293,14 +299,14 @@ class FanOut(Generic[_Item]): plan : FanOutPlan The plan being driven (read-only after construction). fetch : Callable - The async per-sub-request fetch function. + The async per-chunk fetch function. finalize : Callable Transform applied to the combined result (see :data:`_Finalize`) at the terminal :meth:`_run` return, so a completed call yields the caller's finished shape. The ``partial_*`` accessors deliberately skip it and stay raw. partial_frame : pandas.DataFrame - Raw combined frame of completed sub-requests (live; recomputed per + Raw combined frame of completed chunks (live; recomputed per access). Not finalized — call :meth:`resume` for the finished shape. partial_response : httpx.Response or None Raw aggregate response (canonical URL restored), or ``None`` when @@ -309,8 +315,8 @@ class FanOut(Generic[_Item]): def __init__( self, - plan: FanOutPlan[_Item], - fetch: _Fetch[_Item], + plan: FanOutPlan[_Chunk], + fetch: _Fetch[_Chunk], retry_policy: RetryPolicy = _NO_RETRY, finalize: _Finalize = _passthrough_result, client_options: dict[str, Any] | None = None, @@ -345,14 +351,14 @@ def __init__( # reporter). :meth:`resume` runs every drive inside this snapshot, so # a *later* ``exc.call.resume()`` — which fires after those ``with`` # blocks have exited and reset their ContextVars — still rebuilds - # sub-requests against the original API's base URL/dialect rather than + # chunks against the original API's base URL/dialect rather than # the process defaults. The adapter's request builder reads those - # ContextVars when it reconstructs each sub-request, so the snapshot + # ContextVars when it reconstructs each chunk, so the snapshot # must outlive them. The mechanism is generic; which ambients matter is # the adapter's business. self._ctx = copy_context() # Completed (frame, response) pairs keyed by sub-args index; sparse - # (gathered sub-requests complete out of order — see class docstring). + # (gathered chunks complete out of order — see class docstring). # ``_run``'s ``track`` closure is the only writer, so ``dict`` insertion # order is completion order (relied on by :meth:`_combine_raw`). self._chunks: dict[int, tuple[pd.DataFrame, httpx.Response]] = {} @@ -369,7 +375,7 @@ def wrap_failure(self, exc: BaseException) -> FanOutInterrupted | None: Parameters ---------- exc : BaseException - The exception raised by a sub-request. + The exception raised by a chunk. Returns ------- @@ -402,11 +408,11 @@ def _normalize_failure(self, exc: BaseException) -> BaseException: @property def completed_chunks(self) -> int: - """Number of sub-requests completed so far.""" + """Number of chunks completed so far.""" return len(self._chunks) def _combine_raw(self) -> tuple[pd.DataFrame, httpx.Response]: - """Assemble the raw ``(frame, response)`` from completed sub-requests, + """Assemble the raw ``(frame, response)`` from completed chunks, before :attr:`finalize` runs. Frames concatenate in sub-args *index* order (``sorted`` keys — @@ -426,7 +432,7 @@ def _combine_raw(self) -> tuple[pd.DataFrame, httpx.Response]: return self._combine_frames(), self._combine_responses() def _combine_frames(self) -> pd.DataFrame: - """Combine completed frames in deterministic sub-request order.""" + """Combine completed frames in deterministic chunk order.""" return _combine_chunk_frames([self._chunks[i][0] for i in sorted(self._chunks)]) def _combine_responses(self) -> httpx.Response: @@ -437,7 +443,7 @@ def _combine_responses(self) -> httpx.Response: @property def partial_frame(self) -> pd.DataFrame: """ - Raw combined frame of sub-requests that have completed so far. + Raw combined frame of chunks that have completed so far. Live — recomputed on each access so it reflects current state across resume attempts. Deliberately the *raw* combined frame @@ -451,7 +457,7 @@ def partial_frame(self) -> pd.DataFrame: Returns ------- pandas.DataFrame - Combined frame of completed sub-requests, or an empty + Combined frame of completed chunks, or an empty ``DataFrame`` when nothing has completed. """ return self._combine_frames() if self._chunks else pd.DataFrame() @@ -469,18 +475,18 @@ def partial_response(self) -> httpx.Response | None: Returns ------- httpx.Response or None - Aggregated response when at least one sub-request has + Aggregated response when at least one chunk has completed, ``None`` otherwise. """ return self._combine_responses() if self._chunks else None - def _pending(self) -> Iterator[tuple[int, _Item]]: + def _pending(self) -> Iterator[tuple[int, _Chunk]]: """ - Yield ``(index, item)`` for sub-requests not yet completed. + Yield ``(index, item)`` for chunks not yet completed. Iterates the plan in its deterministic order and skips any index already in ``self._chunks``. :meth:`_run` uses this to pick up - exactly the sub-requests it still owes — the mechanism behind + exactly the chunks it still owes — the mechanism behind idempotent resume. """ for index, item in enumerate(self.plan): @@ -496,7 +502,7 @@ def resume(self) -> tuple[pd.DataFrame, Any]: works whether or not the caller is already inside an event loop (Jupyter / IPython / async apps). The portal copies the calling context, so the active progress reporter still reaches the - sub-requests. + chunks. This executor is what emits progress events, so it is also what owns the reporter's lifetime: an adapter that drives a ``FanOut`` gets the @@ -504,7 +510,7 @@ def resume(self) -> tuple[pd.DataFrame, Any]: ``with progress_context(...)`` block. A reporter already active (a nested getter, or a caller's own context) is reused unchanged. - Idempotent: only sub-requests whose index isn't already in + Idempotent: only chunks whose index isn't already in ``self._chunks`` are re-issued. Item order is the plan's own and is deterministic, so a partial completion (sparse indices) resumes correctly. @@ -512,7 +518,7 @@ def resume(self) -> tuple[pd.DataFrame, Any]: Returns ------- df : pandas.DataFrame - Combined data from every successful sub-request. + Combined data from every successful chunk. response The finalized aggregate — a raw :class:`httpx.Response` (canonical URL, headers from the response with the lowest reported @@ -540,7 +546,7 @@ def resume(self) -> tuple[pd.DataFrame, Any]: # ``__init__``). ``start_blocking_portal`` copies the *calling* context # into its worker thread, and running here means that calling context # is the snapshot — so the base URL / dialect / row cap active when the - # call was created reach the rebuilt sub-requests, even when this is a + # call was created reach the rebuilt chunks, even when this is a # resume fired long after the original ``with`` blocks exited. The # reporter is the one ambient that must NOT come from the snapshot: a # reporter captured there belongs to a context that has since closed @@ -565,12 +571,12 @@ def _resume_in_context( async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: """ - Gather every pending sub-request over one shared + Gather every pending chunk over one shared :class:`httpx.AsyncClient` and return the combined, finalized result. - Pending sub-requests (:meth:`_pending`) fan out under + Pending chunks (:meth:`_pending`) fan out under ``asyncio.gather`` with ``return_exceptions=True`` so completed - sub-requests survive a sibling's transient failure. On a + chunks survive a sibling's transient failure. On a recognized transient (:class:`~dataretrieval.exceptions.RateLimited`, :class:`~dataretrieval.exceptions.ServiceUnavailable`, or a bare ``httpx.HTTPError`` / ``httpx.InvalidURL``) a @@ -578,7 +584,7 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: ``.call``; ``exc.call.resume()`` then re-issues only the unfinished indices through this same runner. - The gather dispatches *every* pending sub-request at once, but an + The gather dispatches *every* pending chunk at once, but an ``asyncio.Semaphore`` caps the number of concurrent fetches at ``N = max_concurrent`` — ``None`` lifts the cap, ``N=1`` runs them one at a time. The connection pool is sized to the same ``N`` @@ -586,14 +592,14 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: so the in-flight fetches reuse keepalive connections. The semaphore, not the pool, is deliberately the throttle. If the - pool throttled instead, the excess sub-requests would queue + pool throttled instead, the excess chunks would queue *inside* httpx waiting for a connection, and that wait counts against the pool-acquire timeout (60 s, from ``HTTPX_ASYNC_DEFAULTS``). A batch of slow pages that keeps every connection busy past that window would then trip ``httpx.PoolTimeout`` on the queued tail — a purely client-side failure that consumes the retry budget and surfaces as a spurious resumable ``ServiceInterrupted``. Holding - sub-requests at the semaphore keeps them out of the pool until a + chunks at the semaphore keeps them out of the pool until a slot frees, so the pool timeout only fires for a genuinely stuck connection. @@ -603,13 +609,13 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: Parameters ---------- max_concurrent : int or None - Maximum sub-requests in flight (the semaphore value, and the + Maximum chunks in flight (the semaphore value, and the connection-pool size). ``None`` lifts the cap entirely. Returns ------- df : pandas.DataFrame - Combined data from every sub-request. + Combined data from every chunk. response The finalized aggregate — a raw :class:`httpx.Response` (canonical URL, headers from the response with the lowest reported @@ -619,8 +625,8 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: Raises ------ FanOutInterrupted - On a transient sub-request failure. ``.call`` is ``self``, - holding the sparse completed sub-requests; ``.call.resume()`` + On a transient chunk failure. ``.call`` is ``self``, + holding the sparse completed chunks; ``.call.resume()`` re-issues the unfinished ones. """ # The semaphore is the throttle; the pool is merely sized to match @@ -644,9 +650,9 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: reporter.set_chunks(len(self.plan)) async def track( - index: int, item: _Item + index: int, item: _Chunk ) -> tuple[pd.DataFrame, httpx.Response]: - """One sub-request (with retry) + result-store + progress tick.""" + """One chunk (with retry) + result-store + progress tick.""" result = await _retry( lambda: self.fetch(item), self.retry_policy, gate=semaphore ) @@ -657,7 +663,7 @@ async def track( reporter.start_chunk(self.completed_chunks) return result - # Dispatch every pending sub-request concurrently; the + # Dispatch every pending chunk concurrently; the # semaphore (held by ``_retry`` per attempt) is the only throttle. # ``return_exceptions`` keeps completed pairs after a sibling # fails, so partial state stays recoverable via :meth:`resume`. @@ -685,8 +691,8 @@ async def track( # only the first transient is ever raised. Asking # ``wrap_failure`` per failure would snapshot the combined # frame N times (a full concat over every completed - # sub-request) and discard all but one, which a batch of - # sub-requests failing together makes routine. + # chunk) and discard all but one, which a batch of + # chunks failing together makes routine. first_transient: BaseException | None = None for exc in failures: if _classify_chunk_error(exc) is None: diff --git a/dataretrieval/transport/liveness.py b/dataretrieval/transport/liveness.py index 26d78936..983e82e1 100644 --- a/dataretrieval/transport/liveness.py +++ b/dataretrieval/transport/liveness.py @@ -8,7 +8,7 @@ of liveness (a streaming body reader, a chunk-level fetch) somewhere to report. The stamp lives in a :class:`~contextvars.ContextVar` so concurrent retrievals -- -each sub-request of a chunked call, each location of a Water Use fan-out -- +each chunk of a chunked call, each location of a Water Use fan-out -- measure their own silence instead of sharing one clock. """ diff --git a/dataretrieval/transport/retry.py b/dataretrieval/transport/retry.py index 9bf3a6ed..95dd1b88 100644 --- a/dataretrieval/transport/retry.py +++ b/dataretrieval/transport/retry.py @@ -43,7 +43,7 @@ _RETRY_BASE_BACKOFF = 0.5 _RETRY_MAX_BACKOFF = 30.0 _RETRY_AFTER_CAP = 60.0 -# Most a server-named delay is nudged by, to keep sub-requests handed the same +# Most a server-named delay is nudged by, to keep chunks handed the same # hint from waking together. Small on purpose: the server named the wait, so # jitter here decorrelates rather than extends it. _RETRY_AFTER_JITTER = 1.0 @@ -182,7 +182,7 @@ def backoff(self, attempt: int, retry_after: float | None) -> float: A jittered component is always included, even when the server named a delay: a hint of ``0`` -- or a ``Retry-After`` date that has already passed -- would otherwise become a zero-delay re-send against a service - that just asked us to slow down, and sub-requests handed the same hint + that just asked us to slow down, and chunks handed the same hint would all wake at the same instant and burst together. On a server hint that jitter is a small decorrelating nudge rather than diff --git a/dataretrieval/waterdata/cql.py b/dataretrieval/waterdata/cql.py index f83a1f2b..5139b071 100644 --- a/dataretrieval/waterdata/cql.py +++ b/dataretrieval/waterdata/cql.py @@ -24,20 +24,30 @@ from dataretrieval.ogc.shaping import _finalize_ogc from dataretrieval.waterdata.utils import ( _EXTRA_ID_COLS, - _OUTPUT_ID_BY_SERVICE, + _OUTPUT_ID_BY_COLLECTION, OGC_API_URL, WATERDATA_DIALECT, + _accept_legacy_kwargs, ) if TYPE_CHECKING: from dataretrieval._response_metadata import BaseMetadata from dataretrieval.waterdata.types import ( - WATERDATA_SERVICES, + WATERDATA_COLLECTIONS, ) +@_accept_legacy_kwargs( + {"service": "collection"}, + detail=( + "OGC API - Features names this value the collectionId (17-069r4 " + "Requirements 18 and 20, /collections/{id}/items), while `service` " + "names the API itself (Water Data, NGWMN). `service` will be removed " + "on or after 2027-08-09." + ), +) def get_cql( - service: WATERDATA_SERVICES, + collection: WATERDATA_COLLECTIONS, cql: str | dict[str, Any], *, properties: str | Iterable[str] | None = None, @@ -48,9 +58,9 @@ def get_cql( ) -> tuple[pd.DataFrame, BaseMetadata]: """Query a Water Data OGC API collection with an arbitrary CQL2 filter. - Sends ``cql`` as a CQL2 filter against ``service`` and returns the matching + Sends ``cql`` as a CQL2 filter against ``collection`` and returns the matching features, shaped like the typed getters (``get_daily``, ``get_continuous``, - …): the wire ``id`` renamed to the service's id column, columns ordered and + …): the wire ``id`` renamed to the collection's id column, columns ordered and sorted, and dtypes coerced. Use it when you need a predicate the typed getters can't express — a top-level ``or``, ``like`` with ``%`` wildcards, comparison operators, nested boolean trees, or a geometry predicate beyond a @@ -66,9 +76,9 @@ def get_cql( Parameters ---------- - service : str + collection : str OGC collection name. Must be one of - :data:`dataretrieval.waterdata.types.WATERDATA_SERVICES` + :data:`dataretrieval.waterdata.types.WATERDATA_COLLECTIONS` (e.g. ``"daily"``, ``"monitoring-locations"``). cql : str or dict CQL2 query. A ``dict`` is JSON-serialized for transport; a ``str`` is @@ -76,7 +86,7 @@ def get_cql( ``Content-Type: application/query-cql-json``. properties : str or iterable of str, optional Server-side property whitelist (passed as ``properties=`` on the URL). - Reduces payload size. ``"id"`` resolves to the service's ``output_id`` + Reduces payload size. ``"id"`` resolves to the collection's ``output_id`` (e.g. ``daily_id``) the same way it does in the typed wrappers. bbox : list of float, optional Bounding box ``[xmin, ymin, xmax, ymax]`` in CRS 4326. Combines with the @@ -124,23 +134,23 @@ def get_cql( ... }, ... ], ... } - >>> df, md = waterdata.get_cql(service="daily", cql=cql) + >>> df, md = waterdata.get_cql(collection="daily", cql=cql) >>> # Monitoring locations whose HUC starts with "02070010" >>> # (LIKE with the CQL2 ``%`` wildcard). >>> df, md = waterdata.get_cql( - ... service="monitoring-locations", + ... collection="monitoring-locations", ... cql='{"op": "like", "args": [' ... '{"property": "hydrologic_unit_code"},' ... ' "02070010%"]}', ... ) """ - if service not in _OUTPUT_ID_BY_SERVICE: + if collection not in _OUTPUT_ID_BY_COLLECTION: raise ValueError( - f"Unknown service {service!r}. Valid services: " - f"{sorted(_OUTPUT_ID_BY_SERVICE)}." + f"Unknown collection {collection!r}. Valid collections: " + f"{sorted(_OUTPUT_ID_BY_COLLECTION)}." ) - output_id = _OUTPUT_ID_BY_SERVICE[service] + output_id = _OUTPUT_ID_BY_COLLECTION[collection] # ``dict`` is the pythonic input — serialize on the way out. ``str`` is sent # verbatim so callers who already have a CQL2 doc (e.g. imported from a @@ -152,14 +162,14 @@ def get_cql( # Drop id aliases (``daily_id``/``id``) and ``geometry`` from the wire # ``properties`` (the feature ``id`` is always returned and renamed # downstream), matching the typed getters. - wire_properties = _switch_properties_id(properties_list, output_id, service) + wire_properties = _switch_properties_id(properties_list, output_id, collection) - # The OGC package names no service of its own, so this hand-built request + # The OGC package names no collection of its own, so this hand-built request # path states the target itself -- request construction and the empty-result # schema lookup in ``_finalize_ogc`` both read the base URL from here. with _ogc_base_url(OGC_API_URL): req = _construct_cql_request( - service, + collection, body, properties=wire_properties, bbox=bbox, @@ -167,7 +177,7 @@ def get_cql( skip_geometry=skip_geometry, ) - df, response = fetch_ogc_request(req, service=service) + df, response = fetch_ogc_request(req, collection=collection) return _finalize_ogc( df, @@ -175,7 +185,7 @@ def get_cql( properties=properties_list, output_id=output_id, convert_type=convert_type, - service=service, + collection=collection, extra_id_cols=_EXTRA_ID_COLS, dialect=WATERDATA_DIALECT, ) diff --git a/dataretrieval/waterdata/measurements.py b/dataretrieval/waterdata/measurements.py index e3aec503..ddbe578e 100644 --- a/dataretrieval/waterdata/measurements.py +++ b/dataretrieval/waterdata/measurements.py @@ -220,12 +220,12 @@ def get_field_measurements( ... time="P20Y", ... ) """ - service = "field-measurements" + collection = "field-measurements" # Build argument dictionary, omitting None values args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_peaks( @@ -361,11 +361,11 @@ def get_peaks( ... ) """ - service = "peaks" + collection = "peaks" args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_channel( @@ -561,11 +561,11 @@ def get_channel( ... monitoring_location_id="USGS-02238500", ... ) """ - service = "channel-measurements" + collection = "channel-measurements" args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) __all__ = ["get_field_measurements", "get_peaks", "get_channel"] diff --git a/dataretrieval/waterdata/metadata.py b/dataretrieval/waterdata/metadata.py index c3e88ea0..6ab681e2 100644 --- a/dataretrieval/waterdata/metadata.py +++ b/dataretrieval/waterdata/metadata.py @@ -332,7 +332,7 @@ def get_monitoring_locations( ... properties=["monitoring_location_id", "state_name", "country_name"], ... ) """ - service = "monitoring-locations" + collection = "monitoring-locations" # Build argument dictionary, omitting None values (resolving the unified # `state` argument into the OGC `state_name` queryable). @@ -340,7 +340,7 @@ def get_monitoring_locations( _with_state(locals(), to="name", into="state_name"), exclude={"max_rows"} ) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_time_series_metadata( @@ -583,7 +583,7 @@ def get_time_series_metadata( ... begin="1990-01-01/..", ... ) """ - service = "time-series-metadata" + collection = "time-series-metadata" # Build argument dictionary, omitting None values (resolving the unified # `state` argument into the OGC `state_name` queryable). @@ -591,7 +591,7 @@ def get_time_series_metadata( _with_state(locals(), to="name", into="state_name"), exclude={"max_rows"} ) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_combined_metadata( @@ -842,14 +842,14 @@ def get_combined_metadata( ... ) """ - service = "combined-metadata" + collection = "combined-metadata" # Resolve the unified `state` argument into the OGC `state_name` queryable. args = _get_args( _with_state(locals(), to="name", into="state_name"), exclude={"max_rows"} ) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_field_measurements_metadata( @@ -980,11 +980,11 @@ def get_field_measurements_metadata( ... ) """ - service = "field-measurements-metadata" + collection = "field-measurements-metadata" args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) __all__ = [ diff --git a/dataretrieval/waterdata/reference.py b/dataretrieval/waterdata/reference.py index 7db5fb80..14a1884b 100644 --- a/dataretrieval/waterdata/reference.py +++ b/dataretrieval/waterdata/reference.py @@ -112,7 +112,7 @@ def get_reference_table( if limit is not None: query_args["limit"] = limit return get_ogc_data( - args=query_args, output_id=output_id, service=collection, max_rows=max_rows + args=query_args, output_id=output_id, collection=collection, max_rows=max_rows ) diff --git a/dataretrieval/waterdata/time_series.py b/dataretrieval/waterdata/time_series.py index 75594549..f3c7b572 100644 --- a/dataretrieval/waterdata/time_series.py +++ b/dataretrieval/waterdata/time_series.py @@ -236,7 +236,7 @@ def get_daily( >>> # Chain queries: pull all stream sites in a state, then their >>> # daily discharge for the last week. The site list can be hundreds >>> # of values long — the request is transparently chunked across - >>> # multiple sub-requests so the URL stays under the server's byte + >>> # multiple chunks so the URL stays under the server's byte >>> # limit. Combined output looks like a single query. >>> sites_df, _ = dataretrieval.waterdata.get_monitoring_locations( ... state="Ohio", @@ -248,12 +248,12 @@ def get_daily( ... time="P7D", ... ) """ - service = "daily" + collection = "daily" # Build argument dictionary, omitting None values args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_continuous( @@ -446,12 +446,12 @@ def get_continuous( ... filter_lang="cql-text", ... ) """ - service = "continuous" + collection = "continuous" # Build argument dictionary, omitting None values args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_latest_continuous( @@ -657,12 +657,12 @@ def get_latest_continuous( ... monitoring_location_id=["USGS-05114000", "USGS-09423350"] ... ) """ - service = "latest-continuous" + collection = "latest-continuous" # Build argument dictionary, omitting None values args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_latest_daily( @@ -869,12 +869,12 @@ def get_latest_daily( ... monitoring_location_id=["USGS-05114000", "USGS-09423350"] ... ) """ - service = "latest-daily" + collection = "latest-daily" # Build argument dictionary, omitting None values args = _get_args(locals(), exclude={"max_rows"}) - return get_ogc_data(args, service, max_rows=max_rows) + return get_ogc_data(args, collection, max_rows=max_rows) def get_stats_por( diff --git a/dataretrieval/waterdata/types.py b/dataretrieval/waterdata/types.py index ff13f42f..65a61d48 100644 --- a/dataretrieval/waterdata/types.py +++ b/dataretrieval/waterdata/types.py @@ -4,6 +4,7 @@ "CODE_SERVICES", "METADATA_COLLECTIONS", "SERVICES", + "WATERDATA_COLLECTIONS", "WATERDATA_SERVICES", "PROFILES", "PROFILE_LOOKUP", @@ -51,10 +52,10 @@ ] # OGC API time-series/monitoring collections queryable via ``get_cql``. -# Keep in sync with ``utils._OUTPUT_ID_BY_SERVICE`` (same keys): that dict maps +# Keep in sync with ``utils._OUTPUT_ID_BY_COLLECTION`` (same keys): that dict maps # each service to its user-facing ``id`` column and is the runtime source of # truth ``get_cql`` validates against. -WATERDATA_SERVICES = Literal[ +WATERDATA_COLLECTIONS = Literal[ "channel-measurements", "combined-metadata", "continuous", @@ -68,6 +69,11 @@ "time-series-metadata", ] +#: Permanent alias. OGC API - Features calls these collections -- the value is +#: the ``collectionId`` in ``/collections/{id}/items`` -- but this name is the one +#: the package published first, so it keeps resolving. +WATERDATA_SERVICES = WATERDATA_COLLECTIONS + PROFILES = Literal[ "actgroup", "actmetric", diff --git a/dataretrieval/waterdata/utils.py b/dataretrieval/waterdata/utils.py index dfacee5e..c24624d0 100644 --- a/dataretrieval/waterdata/utils.py +++ b/dataretrieval/waterdata/utils.py @@ -1,7 +1,7 @@ """Water Data API layer over the generic OGC facade. This module is the Water-Data-specific adapter: it supplies the -service-to-id map, the CQL2/date-only dialect, and a +collection-to-id map, the CQL2/date-only dialect, and a thin ``get_ogc_data`` wrapper that injects the Water Data defaults. The statistics path lives in its own :mod:`dataretrieval.waterdata.stats` module. @@ -27,19 +27,19 @@ from dataretrieval.ogc import OgcDialect, prepare_request_args from dataretrieval.ogc import get_ogc_data as _facade_get_ogc_data -# Endpoint constants live in one place for the whole service; they are re-bound +# Endpoint constants live in one place for the whole collection; they are re-bound # here because ``waterdata.utils.OGC_API_URL`` is a documented path. from dataretrieval.waterdata.endpoints import BASE_URL, OGC_API_URL, SAMPLES_URL if TYPE_CHECKING: from dataretrieval._response_metadata import BaseMetadata -# Maps each OGC waterdata service to its user-facing ``id`` column (the name the +# Maps each OGC waterdata collection to its user-facing ``id`` column (the name the # typed getters rename the wire ``id`` to, e.g. ``daily`` -> ``daily_id``). -# ``get_cql`` validates its ``service`` argument against these keys and +# ``get_cql`` validates its ``collection`` argument against these keys and # uses the value as the ``output_id`` for result shaping. Keep in sync with the # ``types.WATERDATA_SERVICES`` Literal (same keys). -_OUTPUT_ID_BY_SERVICE: dict[str, str] = { +_OUTPUT_ID_BY_COLLECTION: dict[str, str] = { "channel-measurements": "channel_measurements_id", "combined-metadata": "combined_meta_id", "continuous": "continuous_id", @@ -53,13 +53,14 @@ "time-series-metadata": "time_series_id", } -# Every service's output id EXCEPT the two that are genuinely user-facing +# Every collection's output id EXCEPT the two that are genuinely user-facing # (``monitoring_location_id`` and ``time_series_id``). The rest are synthetic # per-record ids that ``_arrange_cols`` moves to the end of a result frame. -# Derived from ``_OUTPUT_ID_BY_SERVICE`` so adding a service can't silently +# Derived from ``_OUTPUT_ID_BY_COLLECTION`` so adding a collection can't silently # leave a stray id column at the front again. _EXTRA_ID_COLS = frozenset( - set(_OUTPUT_ID_BY_SERVICE.values()) - {"monitoring_location_id", "time_series_id"} + set(_OUTPUT_ID_BY_COLLECTION.values()) + - {"monitoring_location_id", "time_series_id"} ) # The Water Data API dialect: ``monitoring-locations`` doesn't accept @@ -123,7 +124,7 @@ def _flatten_queryables(local_vars: dict[str, Any]) -> dict[str, Any]: ``state_name="Wisconsin"`` is normalized, mutual-exclusion-checked, and sent exactly like a named param. See :func:`dataretrieval.waterdata.get_queryables` for each collection's - filterable properties (the service rejects an unknown one with a 400). + filterable properties (the collection rejects an unknown one with a 400). ``**queryables`` always arrives as a dict (empty when unused) and the key is popped, so this is a no-op on getters without the passthrough and idempotent @@ -173,13 +174,13 @@ def _with_state(local_vars: dict[str, Any], *, to: str, into: str) -> dict[str, def get_ogc_data( args: dict[str, Any], - service: str, + collection: str, output_id: str | None = None, max_rows: int | None = None, ) -> tuple[pd.DataFrame, BaseMetadata]: """Water-Data wrapper over :func:`~dataretrieval.ogc.get_ogc_data`. - Defaults ``output_id`` from the Water Data service map when not given, + Defaults ``output_id`` from the Water Data collection map when not given, and supplies the Water Data extra-id columns and dialect, so the typed getters in ``api.py`` call this unchanged. (Sibling OGC APIs such as NGWMN call ``dataretrieval.ogc.get_ogc_data`` directly with their own @@ -188,12 +189,12 @@ def get_ogc_data( Parameters ---------- args : Dict[str, Any] - Dictionary of request arguments for the OGC service. - service : str + Dictionary of request arguments for the OGC collection. + collection : str The OGC API collection name (e.g., ``"daily"``). output_id : str, optional The user-facing id column the wire ``id`` is renamed to. Defaults - to ``_OUTPUT_ID_BY_SERVICE[service]``; pass it explicitly only for + to ``_OUTPUT_ID_BY_COLLECTION[collection]``; pass it explicitly only for collections outside that map (e.g. reference-table collections). max_rows : int, optional Stop paginating once this many rows have been collected and @@ -209,10 +210,10 @@ def get_ogc_data( query time. """ if output_id is None: - output_id = _OUTPUT_ID_BY_SERVICE[service] + output_id = _OUTPUT_ID_BY_COLLECTION[collection] return _facade_get_ogc_data( args, - service, + collection, output_id, max_rows=max_rows, base_url=OGC_API_URL, @@ -226,6 +227,8 @@ def get_ogc_data( def _accept_legacy_kwargs( mapping: Mapping[str, str], + *, + detail: str = "", ) -> Callable[[Callable[..., _R]], Callable[..., _R]]: """Accept deprecated keyword-argument names on the decorated function. @@ -243,6 +246,11 @@ def _accept_legacy_kwargs( intentionally relaxed (the wrapper accepts the extra deprecated names), so static checkers won't flag legacy call sites. + ``detail`` appends a sentence to the warning. The default message says only + that the name changed; a rename with a reason worth giving -- a spec that + names the value differently, a removal date -- passes it here rather than + hand-rolling the whole shim to carry one sentence. + Raises ------ TypeError @@ -262,9 +270,12 @@ def wrapper(*args: Any, **kwargs: Any) -> _R: f"{func.__name__}() received both {old_name!r} " f"(deprecated) and {new_name!r}; pass only {new_name!r}." ) - warnings.warn( + message = ( f"The {old_name!r} argument is deprecated and will be " - f"removed in a future release; use {new_name!r} instead.", + f"removed in a future release; use {new_name!r} instead." + ) + warnings.warn( + f"{message} {detail}" if detail else message, DeprecationWarning, stacklevel=2, ) @@ -283,7 +294,7 @@ def wrapper(*args: Any, **kwargs: Any) -> _R: "WATERDATA_DIALECT", "_EXTRA_ID_COLS", "_NO_NORMALIZE_PARAMS", - "_OUTPUT_ID_BY_SERVICE", + "_OUTPUT_ID_BY_COLLECTION", "_accept_legacy_kwargs", "_get_args", "_with_state", diff --git a/dataretrieval/wateruse.py b/dataretrieval/wateruse.py index 2ab24be2..7fe69e59 100644 --- a/dataretrieval/wateruse.py +++ b/dataretrieval/wateruse.py @@ -350,7 +350,7 @@ def _fan_out( precedence, progress, and resumable interruption -- belongs to :class:`~dataretrieval.transport.fanout.FanOut`, which Water Data and NGWMN drive too. This function is now only the NWDC-specific half: what a - sub-request is, and how to read one. + chunk is, and how to read one. The plan is the request list itself. ``FanOut`` asks a plan only to be sized and iterable, and the NWDC accepts one ``location=`` per request, so diff --git a/docs/source/architecture/decisions/0004-error-retry-resume.rst b/docs/source/architecture/decisions/0004-error-retry-resume.rst index d05d7522..70202c65 100644 --- a/docs/source/architecture/decisions/0004-error-retry-resume.rst +++ b/docs/source/architecture/decisions/0004-error-retry-resume.rst @@ -24,7 +24,7 @@ All request failures exposed by public service modules derive from Where automatic recovery is supported, retries are bounded, use exponential backoff with full jitter, honor only bounded ``Retry-After`` delays, and preserve -cancellation. OGC fan-out retains completed subrequests and raises a typed +cancellation. OGC fan-out retains completed chunks and raises a typed ``ChunkInterrupted`` with a handle that resumes only missing work. Fatal or unknown failures are not disguised as resumable transients. diff --git a/docs/source/architecture/decisions/0008-fan-out-execution.rst b/docs/source/architecture/decisions/0008-fan-out-execution.rst index c336d143..d5431f1e 100644 --- a/docs/source/architecture/decisions/0008-fan-out-execution.rst +++ b/docs/source/architecture/decisions/0008-fan-out-execution.rst @@ -49,25 +49,25 @@ completion tracking, and resume. It names no protocol concept. An adapter supplies a ``FanOutPlan`` and an ``async def fetch(item) -> (df, response)``. ``FanOutPlan`` is a ``Protocol`` of ``__len__`` and ``__iter__``, generic in the -item type -- a sized, iterable collection of sub-request descriptions, and +item type -- a sized, iterable collection of chunk descriptions, and nothing more. The executor passes each item to the adapter's own ``fetch`` without inspecting it, so the item type is the adapter's business: the OGC getters yield kwargs dicts, Water Use yields ready ``httpx.Request`` objects. The standard protocols, rather than bespoke members, are a deliberate choice. -A plan declaring ``total`` and ``iter_sub_args()`` would be stating ``len`` +A plan declaring ``total`` and ``iter_chunk_args()`` would be stating ``len`` twice under a private name: the two could then report different counts, and a -test would have to assert they agree. Every adapter whose sub-requests are +test would have to assert they agree. Every adapter whose chunks are already a list would also need a wrapper class whose only job is renaming ``len``. With the standard names a plain ``list`` is a plan, which is exactly what Water -Use passes. ``ChunkPlan`` keeps ``total`` and ``iter_sub_args`` as its own +Use passes. ``ChunkPlan`` keeps ``total`` and ``iter_chunk_args`` as its own vocabulary and defines the dunders to delegate to them, so the two cannot disagree. The protocol is structural rather than nominal for the original reason: -``ChunkPlan`` derives sub-requests from a byte budget over multi-value axes, +``ChunkPlan`` derives chunks from a byte budget over multi-value axes, a list of requests derives nothing, and so there is no shared implementation an abstract base could hold. @@ -85,8 +85,9 @@ The interruption taxonomy moves to ``dataretrieval.interruptions``, a top-level leaf, for the reason ADR 0006 gives for ``combining``, ``progress``, and ``credentials``: adapters need it whether or not they went through transport, and an exception taxonomy is not HTTP execution policy. Its base class is -renamed ``FanOutInterrupted``, since Water Use raises it without chunking -anything. ``ChunkInterrupted`` is retained as a permanent alias of the same +renamed ``FanOutInterrupted`` because the failure interrupts a query's +*execution* rather than its division -- both Water Data and Water Use chunk, +for different reasons, but what fails is the fan-out. ``ChunkInterrupted`` is retained as a permanent alias of the same class object -- not a shim scheduled for deletion -- because it is the name published in the user guide and caught in user code. The subclasses (``QuotaExhausted``, ``ServiceInterrupted``) were already neutral and are @@ -133,7 +134,7 @@ Compliance contains no ``asyncio.gather``, ``Semaphore``, or ``TaskGroup``, so the duplication cannot return. That both plan types are sized and *repeatably* iterable -- resume keys completed work by position, so a generator mistaken for -a collection would re-issue the wrong sub-requests. And that an interruption +a collection would re-issue the wrong chunks. And that an interruption taxonomy does not reappear inside ``transport``. Adapter tests cover Water Use resume re-issuing only unfinished locations, diff --git a/docs/source/architecture/index.rst b/docs/source/architecture/index.rst index b58f4167..a92f19c4 100644 --- a/docs/source/architecture/index.rst +++ b/docs/source/architecture/index.rst @@ -246,14 +246,14 @@ A typical OGC-backed call follows this sequence:: -> normalize and validate arguments -> select OGC dialect and build a chunk plan -> enter a short-lived anyio blocking portal - -> execute subrequests through a shared httpx.AsyncClient - -> paginate each subrequest + -> execute chunks through a shared httpx.AsyncClient + -> paginate each chunk -> retry bounded transient failures -> combine and deduplicate pages/chunks -> shape columns and types -> return DataFrame and BaseMetadata -A transient failure after some subrequests complete raises a resumable +A transient failure after some chunks complete raises a resumable interruption. The retained ``FanOut`` (available as ``ChunkedCall`` on the OGC compatibility path) reissues only missing work and applies the same finalization path when resumed. Cancellation and non-transient programming errors take @@ -293,7 +293,7 @@ Resource and configuration view complements ``API_USGS_RETRIES``, which caps attempts rather than elapsed time: without this bound, four retries of a request that times out after a minute add up to four silent minutes. Progress restarts the budget — a page - received, or a queued sub-request acquiring its concurrency slot. Neither a + received, or a queued chunk acquiring its concurrency slot. Neither a slow but productive download nor the tail of a wide fan-out is cut short, and an attempt already in flight is never interrupted. This bound never withholds the first retry, so one slow attempt cannot disable retry by @@ -306,7 +306,7 @@ Resource and configuration view retrieval results. ``dataretrieval.transport`` centralizes HTTP timeout, redirect, and -authentication policy. OGC subrequest fan-out and Water Use location +authentication policy. OGC chunk fan-out and Water Use location fan-out retain separate explicit concurrency caps because their upstream costs and request shapes differ. diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index 511d8217..046f7a5d 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -81,7 +81,7 @@ over-large request into chunks, and a Water Use call with several locations becomes one request per location. When a transient failure interrupts one mid-stream, the work already completed is preserved: catch ``FanOutInterrupted`` and call ``exc.call.resume()`` once the condition clears --- only the unfinished sub-requests are re-issued. +-- only the unfinished chunks are re-issued. (``ChunkInterrupted`` is the same class under its original name; either works.) @@ -109,14 +109,14 @@ Chunk a large request more finely ================================= By default the getters split an over-large request only as much as the -server's ~8 KB URL limit forces -- the fewest sub-requests. Because each -sub-request paginates, splitting a large result further costs little or no -extra quota *as long as each sub-request still spans many pages*. (Ten states +server's ~8 KB URL limit forces -- the fewest chunks. Because each +chunk paginates, splitting a large result further costs little or no +extra quota *as long as each chunk still spans many pages*. (Ten states pulled as one request then page nearly as many times as ten per-state requests -would; a split that leaves each sub-request only a page or two adds its partial +would; a split that leaves each chunk only a page or two adds its partial final page.) So if you *know* your pull is large, ask for a finer split with ``parallel_chunks(n)``: you trade roughly the same pages for more, smaller -sub-requests, which gives smoother progress, more even concurrency, and a +chunks, which gives smoother progress, more even concurrency, and a smaller unit of retry/resume. ``parallel_chunks`` is a scoped ``with`` block, so an aggressive setting can't leak into unrelated calls and accidentally spend quota: @@ -131,12 +131,12 @@ quota: ) ``n`` is a positive integer (e.g. ``2``, ``8``, ``32``) -- the number of -sub-requests to fan the call out into; a non-integer or non-positive value -raises ``ValueError`` at the ``with``. ``n`` caps the *total* sub-request count +chunks to fan the call out into; a non-integer or non-positive value +raises ``ValueError`` at the ``with``. ``n`` caps the *total* chunk count across every multi-value argument combined (not per argument), bounded below by what the byte limit already forces and above by how many values there are to split. Several multi-value arguments therefore can't multiply past it, and -``n=1`` asks for no extra fan-out. Each sub-request costs a request against your +``n=1`` asks for no extra fan-out. Each chunk costs a request against your hourly rate limit. How many run *at once* is capped separately by ``API_USGS_CONCURRENT`` (default 32), so an ``n`` beyond that adds quota without adding parallelism -- the useful range is roughly ``2`` up to diff --git a/tests/architecture_test.py b/tests/architecture_test.py index 595e2d30..0fff59ef 100644 --- a/tests/architecture_test.py +++ b/tests/architecture_test.py @@ -547,7 +547,7 @@ def test_fan_out_plans_are_sized_and_repeatably_iterable() -> None: load-bearing and *not* guaranteed by the type is repeatability: resume keys completed work by position, so a plan whose second pass differed -- a generator mistaken for a collection, say -- would re-issue the wrong - sub-requests. + chunks. """ import httpx diff --git a/tests/conftest.py b/tests/conftest.py index dd3381bc..de5cfd36 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -4,7 +4,7 @@ * Relaxes ``pytest-httpx``'s strict-mode flags so unconsumed mocks and unmatched requests don't fail the suite (keeps mocked-URL setup terse). * Pins the chunker env for every test (see ``_pin_chunker_env``), so - sub-request dispatch is deterministic and mocked retries measure attempt + chunk dispatch is deterministic and mocked retries measure attempt counts rather than wall clock. Concurrency and retry tests opt in by re-setting the env vars inside their body via ``monkeypatch.setenv``. """ @@ -39,7 +39,7 @@ def _pin_chunker_env(monkeypatch): Production defaults ``API_USGS_CONCURRENT`` to 32, ``API_USGS_RETRIES`` to 4, and ``API_USGS_STALL_TIMEOUT`` to 60 s. - Pinning ``API_USGS_CONCURRENT=1`` keeps sub-request dispatch + Pinning ``API_USGS_CONCURRENT=1`` keeps chunk dispatch deterministic for the mocked suite, and ``API_USGS_RETRIES=0`` makes a single transient surface immediately rather than be retried. Concurrency and retry tests opt in by overriding the env inside diff --git a/tests/contracts/public_api_test.py b/tests/contracts/public_api_test.py index d5614bf4..53e28aeb 100644 --- a/tests/contracts/public_api_test.py +++ b/tests/contracts/public_api_test.py @@ -56,7 +56,7 @@ "get_codes": ("code_service",), "get_combined_metadata": (), "get_continuous": (), - "get_cql": ("service", "cql"), + "get_cql": ("collection", "cql"), "get_daily": (), "get_field_measurements": (), "get_field_measurements_metadata": (), diff --git a/tests/ngwmn_test.py b/tests/ngwmn_test.py index b34a693b..5921aa1e 100644 --- a/tests/ngwmn_test.py +++ b/tests/ngwmn_test.py @@ -290,9 +290,9 @@ def test_get_sites(httpx_mock): def test_get_sites_skip_geometry(httpx_mock): - """``skip_geometry=True`` is forwarded to the service and the resulting + """``skip_geometry=True`` is forwarded to the collection and the resulting frame has no geometry column.""" - # Same fixture minus the geometry key, which is what the service sends back + # Same fixture minus the geometry key, which is what the collection sends back # when asked to skip it. bare = _collection( [_feature(f["properties"], id_=f["id"]) for f in _SITES["features"]] @@ -321,7 +321,7 @@ def test_get_sites_state_accepts_name_postal_or_fips(httpx_mock): for qs in sent: assert qs["state_name"] == ["Wisconsin"] # The shim rewrites into ``state_name``; raw ``state`` must not leak - # through, or the service would silently ignore it. + # through, or the collection would silently ignore it. assert "state" not in qs @@ -374,7 +374,7 @@ def test_get_water_level(httpx_mock): def test_get_water_level_coerces_dialect_columns(httpx_mock): """The NGWMN dialect coerces ``sample_time`` to datetimes and the depth / - level columns to numbers, even though the service sends all of them as + level columns to numbers, even though the collection sends all of them as strings.""" _mock(httpx_mock, "waterLevelObs", _WATER_LEVELS) @@ -392,13 +392,13 @@ def test_get_water_level_coerces_dialect_columns(httpx_mock): def test_get_water_level_datetime_subsets(httpx_mock): - """A bounded ``datetime`` is forwarded as an OGC interval, so the service + """A bounded ``datetime`` is forwarded as an OGC interval, so the collection returns a subset of the full record rather than us filtering client-side.""" _mock(httpx_mock, "waterLevelObs", _WATER_LEVELS) full, _ = ngwmn.get_water_level(monitoring_location_id=_SITE) httpx_mock.reset() - # What the service would return for the window: the 1978 observation drops. + # What the collection would return for the window: the 1978 observation drops. windowed_body = _collection(_WATER_LEVELS["features"][1:]) _mock(httpx_mock, "waterLevelObs", windowed_body) diff --git a/tests/transport_test.py b/tests/transport_test.py index 8008d9e7..dcd4b5c0 100644 --- a/tests/transport_test.py +++ b/tests/transport_test.py @@ -228,7 +228,7 @@ def test_elapsed_retry_after_still_backs_off() -> None: # zero -- exactly when a hint of 0 would become a zero-delay re-send. assert retry.RetryPolicy(base_backoff=0.0).backoff(attempt=1, retry_after=0.0) > 0.0 # A server-named delay is honored, plus a small decorrelating nudge so - # concurrent sub-requests handed the same hint do not all wake together -- + # concurrent chunks handed the same hint do not all wake together -- # and never enough to push the wait past the policy's own bounds. assert 5.0 < policy.backoff(attempt=1, retry_after=5.0) <= 6.0 # A hint already at the cap is never nudged past it -- the jitter would diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 4b521313..19687869 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -251,11 +251,11 @@ def test_chunk_plan_fans_out_filter_when_list_alone_cannot_fit(): assert any(len(plan.chunks[ax.arg_key]) > 1 for ax in plan.axes) -def test_chunk_plan_minimizes_total_sub_requests(): +def test_chunk_plan_minimizes_total_chunks(): """When both axes need shrinking, picking smaller filter chunks frees URL budget for larger list chunks, and vice versa. The planner should pick the allocation with the *fewest* total - sub-requests, not just the first allocation that fits.""" + chunks, not just the first allocation that fits.""" # 16 short clauses (no inflation under URL encoding so the math is # tractable). Each clause = 5 bytes (e.g. "f='0'"); full filter ≈ # 16*5 + 15*4 = 140 bytes raw. @@ -267,7 +267,7 @@ def test_chunk_plan_minimizes_total_sub_requests(): # Tight limit forces both axes to participate. plan = ChunkPlan(args, _fake_build, url_limit=380) # Plan must beat the bail-floor-style worst case (8 singletons × 16 - # filter chunks = 128 sub-requests) by a healthy margin. + # filter chunks = 128 chunks) by a healthy margin. assert plan.total < 128 @@ -287,13 +287,13 @@ def test_chunk_plan_raises_when_smallest_plan_doesnt_fit(): def test_chunk_plan_passthrough_when_request_fits(): """URL under limit → trivial passthrough plan (no axes, total=1), - and ``iter_sub_args`` yields exactly one sub-args dict equal to + and ``iter_chunk_args`` yields exactly one sub-args dict equal to the original args.""" args = {"monitoring_location_id": ["A", "B", "C"], "limit": 100} plan = ChunkPlan(args, _fake_build, url_limit=8000) assert plan.axes == [] assert plan.total == 1 - subs = list(plan.iter_sub_args()) + subs = list(plan.iter_chunk_args()) assert len(subs) == 1 assert subs[0] == args @@ -347,10 +347,10 @@ async def fetch(args): assert len(pcodes_seen) > 1, "pcodes axis was not split" assert len(stats_seen) > 1, "stats axis was not split" - # Cartesian shape: # sub-requests == product of unique chunks across axes + # Cartesian shape: # chunks == product of unique chunks across axes expected = len(sites_seen) * len(pcodes_seen) * len(stats_seen) assert len(calls) == expected, ( - f"expected {expected} cartesian-product sub-requests, got {len(calls)}" + f"expected {expected} cartesian-product chunks, got {len(calls)}" ) # And no triple repeats (exhaustive enumeration, no duplicates). assert len(set(calls)) == len(calls) @@ -384,11 +384,11 @@ async def fetch(args): assert len(calls) > 1, "patched constant should drive chunking" -def test_chunked_session_shared_across_sub_requests(): - """Every sub-request of one chunked call sees the same +def test_chunked_session_shared_across_chunks(): + """Every chunk of one chunked call sees the same ``httpx.AsyncClient`` on the ``_chunked_client`` ContextVar, so downstream paginated helpers (``_walk_pages``) can reuse the - connection pool instead of handshaking fresh on each sub-request.""" + connection pool instead of handshaking fresh on each chunk.""" sessions_seen = [] @multi_value_chunked(build_request=_fake_build, url_limit=240) @@ -406,7 +406,7 @@ async def fetch(args): # Plan must actually fan out — otherwise the test isn't exercising # the shared-session path. assert len(sessions_seen) > 1 - # Every sub-request saw a Session, not None. + # Every chunk saw a Session, not None. assert all(s is not None for s in sessions_seen) # And it was the same object every time. assert len({id(s) for s in sessions_seen}) == 1 @@ -441,7 +441,7 @@ async def fetch(args): with pytest.raises(QuotaExhausted) as excinfo: fetch({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) - # First run published a shared client to its sub-requests; the calling + # First run published a shared client to its chunks; the calling # thread's ContextVar is unaffected (reads its default). assert _chunked_client.get() is None first_run_sessions = list(sessions_seen) @@ -497,16 +497,16 @@ async def fetch(args): decorated({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10, "S5" * 10]}) err = excinfo.value - # Async fan-out: every non-failing sub-request completes (the gather + # Async fan-out: every non-failing chunk completes (the gather # runs all of them; only i==2 raises), so 4 of 5 complete. - assert err.completed_chunks == 4 # only the i==2 sub-request failed + assert err.completed_chunks == 4 # only the i==2 chunk failed assert err.total_chunks == 5 assert err.partial_frame is not None assert set(err.partial_frame["i"]) == {0, 1, 3, 4} def test_quota_exhausted_on_first_chunk_429_has_no_partial_response(): - """A 429 on the very first sub-request means no responses have + """A 429 on the very first chunk means no responses have completed; ``partial_response`` is ``None`` (and ``partial_frame`` is empty) so callers can branch on that to distinguish "abort before any data arrived" from "abort after partial collection".""" @@ -526,11 +526,11 @@ async def fetch(args): def test_quota_exhausted_resume_picks_up_where_429_stopped(): """After a mid-call 429 ``ChunkedCall`` raises ``QuotaExhausted``; once the window resets, ``e.call.resume()`` re-issues only the - sub-requests that hadn't completed and returns the full combined + chunks that hadn't completed and returns the full combined result. Chunks completed before the 429 are not re-fetched.""" - # One sub-request (the chunk containing the failing site) 429s on the + # One chunk (the chunk containing the failing site) 429s on the # first gather, then succeeds once the window resets. Under the async - # fan-out every OTHER sub-request completes on the first gather, so + # fan-out every OTHER chunk completes on the first gather, so # resume re-issues only the single still-pending chunk. We track which # sub-args have been issued to assert the completed chunks aren't # re-fetched. @@ -553,7 +553,7 @@ async def fetch(args): sites = ["S1" * 10, "S2" * 10, failing_site, "S4" * 10, "S5" * 10] # First attempt: 429 on the chunk carrying the failing site; the other - # four sub-requests complete. + # four chunks complete. with pytest.raises(QuotaExhausted) as excinfo: decorated({"sites": sites}) err = excinfo.value @@ -561,7 +561,7 @@ async def fetch(args): pre_resume_count = len(fetched_sites) assert pre_resume_count == 4 # every chunk but the failing one completed - # Resume: re-issues only the still-pending sub-request. + # Resume: re-issues only the still-pending chunk. df, _ = err.call.resume() # Exactly one more fetch happened on resume (the chunk that 429'd); @@ -581,8 +581,8 @@ def test_quota_exhausted_resume_can_reraise_on_persistent_429(): subsequent resume after a longer wait still picks up cleanly.""" # Key the failure on the chunk's CONTENT (one persistently-429ing # site) rather than a global call counter: under the async fan-out - # every other sub-request completes, and the same still-pending - # sub-request re-fails on resume — so the completed count is stable. + # every other chunk completes, and the same still-pending + # chunk re-fails on resume — so the completed count is stable. failing_site = "S3" * 10 async def fetch(args): @@ -602,7 +602,7 @@ async def fetch(args): first.value.call.resume() # Both exceptions report the same completed_chunks count — every - # sub-request but the persistently-429ing one completed on the first + # chunk but the persistently-429ing one completed on the first # gather, and the resume re-issued only that one, which 429'd again. assert first.value.completed_chunks == 4 assert second.value.completed_chunks == 4 @@ -652,7 +652,7 @@ async def fetch(args): decorated_a = multi_value_chunked(build_request=_fake_build, url_limit=240)(fetch_a) df_a, _ = decorated_a({"sites": sites}) - # Run B: trigger 429 on the third sub-request, then resume. + # Run B: trigger 429 on the third chunk, then resume. fetch_b = make_fetch(rate_limit_at_call=3) decorated_b = multi_value_chunked(build_request=_fake_build, url_limit=240)(fetch_b) with pytest.raises(QuotaExhausted) as excinfo: @@ -677,14 +677,14 @@ async def fetch(args): def test_resume_rebuilds_in_captured_context(): - """Regression: sub-requests are rebuilt by reading ambient ContextVars + """Regression: chunks are rebuilt by reading ambient ContextVars (the engine threads base URL / dialect / row cap that way). A ``call.resume()`` fired AFTER the originating ``with`` block exits — the documented recovery for a mid-stream 429 — must still observe the values active when the call was *created*, not the process defaults. ``ChunkedCall`` snapshots the context at construction and runs every drive inside it; without that snapshot a resumed NGWMN call would - rebuild its sub-requests against the wrong (default Water Data) base.""" + rebuild its chunks against the wrong (default Water Data) base.""" var = contextvars.ContextVar("ctx_probe", default="DEFAULT") observed: list[str] = [] @@ -716,11 +716,11 @@ async def fetch(args): assert var.get() == "DEFAULT" assert 0 < excinfo.value.completed_chunks < excinfo.value.total_chunks - # Resume OUTSIDE the context. Every rebuilt sub-request must still see + # Resume OUTSIDE the context. Every rebuilt chunk must still see # "IN" (the captured snapshot), never the leaked "DEFAULT". observed.clear() df, _ = excinfo.value.call.resume() - assert observed, "resume issued no sub-requests" + assert observed, "resume issued no chunks" assert set(observed) == {"IN"}, observed assert sorted(df["id"].tolist()) == sorted(sites) @@ -750,7 +750,7 @@ def test_chunker_wraps_service_unavailable_as_resumable(): transport failure: ``ChunkedCall`` must wrap it as ``ServiceInterrupted`` carrying the partial state, parallel to how a 429 becomes ``QuotaExhausted``. Once the upstream recovers, - ``.call.resume()`` resumes only the still-pending sub-requests.""" + ``.call.resume()`` resumes only the still-pending chunks.""" state = {"i": 0, "blow_up": True} async def fetch(args): @@ -772,7 +772,7 @@ async def fetch(args): err = excinfo.value # Resumable: handle on .call with already-completed work preserved. assert err.call is not None - # Async fan-out: only the i==2 sub-request fails; the gather completes + # Async fan-out: only the i==2 chunk fails; the gather completes # the other four, so 4 of 5 are recorded before the failure surfaces. assert err.completed_chunks == 4 assert err.total_chunks == 5 @@ -838,7 +838,7 @@ def test_chunk_interrupted_with_partial_data_pickles_intact(): {"monitoring_location_id": ["A", "B", "C"]}, _fake_build, url_limit=8000 ) call = ChunkedCall(plan, lambda args: (pd.DataFrame(), None)) - # One sub-request completed before the failure: a real frame + response. + # One chunk completed before the failure: a real frame + response. call._chunks[0] = ( pd.DataFrame({"id": ["A"]}), httpx.Response( @@ -882,7 +882,7 @@ async def fetch(args): decorated({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10, "S5" * 10]}) err = excinfo.value - # Async fan-out: only the i==2 sub-request fails; the other four complete. + # Async fan-out: only the i==2 chunk fails; the other four complete. assert err.completed_chunks == 4 assert err.call is not None # The transport exception is on __cause__ so callers can drill in if needed. @@ -916,14 +916,14 @@ async def fetch(args): decorated({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10, "S5" * 10]}) err = excinfo.value - # Async fan-out: only the i==2 sub-request fails; the other four complete. + # Async fan-out: only the i==2 chunk fails; the other four complete. assert err.completed_chunks == 4 assert err.call is not None assert isinstance(err.__cause__, httpx.InvalidURL) # The top-level message must surface the underlying cause text so # the user doesn't have to traverse ``__cause__`` to know what # actually failed (previously the message was generic "Service - # error after K/N sub-requests; ... resume() once the upstream + # error after K/N chunks; ... resume() once the upstream # recovers", with the real "URL too long" only visible via # ``.__cause__``). assert "InvalidURL" in str(err) @@ -1018,8 +1018,8 @@ async def fetch(args): _quota_response(500), ) - # 2 sites at url_limit=240 → 2 singleton sub-requests. The 429 fires - # on the SECOND sub-request and the gather completes the other, so the + # 2 sites at url_limit=240 → 2 singleton chunks. The 429 fires + # on the SECOND chunk and the gather completes the other, so the # exception captures exactly ONE completed chunk — the path where # _combine_chunk_frames aliases its single non-empty frame. decorated = multi_value_chunked(build_request=_fake_build, url_limit=240)(fetch) @@ -1060,7 +1060,7 @@ def test_combine_chunk_responses_returns_independent_headers(): def test_combine_chunk_responses_surfaces_lowest_remaining(): - """``x-ratelimit-remaining`` reports the LOWEST any sub-request saw — the + """``x-ratelimit-remaining`` reports the LOWEST any chunk saw — the quota actually left after the fan-out — not the last-by-index, which under concurrency need not be the response the server processed last.""" r0 = mock.Mock( @@ -1230,7 +1230,7 @@ def test_chunk_plan_handles_initial_url_overflow(): """A user query whose unchunked URL exceeds the 64 KB ``httpx.URL`` cap (e.g. 5000+ site IDs comma-joined) must not crash ``ChunkPlan.__init__``; the planner falls back to a - worst-case sub-request URL for ``canonical_url`` and proceeds to + worst-case chunk URL for ``canonical_url`` and proceeds to halve the over-limit axes normally.""" real_build = _fake_build @@ -1261,7 +1261,7 @@ def test_multi_value_chunked_restores_canonical_url(): @multi_value_chunked(build_request=_fake_build, url_limit=240) async def fetch(args): - # Each sub-response carries the chunked sub_args's URL, so + # Each sub-response carries the chunked chunk_args's URL, so # without canonical restoration the first chunk's URL would # leak through to md.url. sub_url = _fake_build(**args).url @@ -1276,9 +1276,9 @@ async def fetch(args): assert len(sub_urls) > 1, "test setup error: chunker didn't fan out" # md.url must equal the URL the unchunked query would have produced. assert md.url == _fake_build(sites=sites).url - # And differ from every sub-request's URL (each carries a smaller list). + # And differ from every chunk's URL (each carries a smaller list). assert all(md.url != u for u in sub_urls) - # The canonical URL is strictly bigger byte-wise than any sub-request. + # The canonical URL is strictly bigger byte-wise than any chunk. assert all(len(md.url) > len(u) for u in sub_urls) @@ -1304,7 +1304,7 @@ def test_extract_axes_skips_scalar_contract_params(): bad cast), ``_extract_axes`` must NOT treat it as a multi-value axis. Chunking ``limit`` would silently fan into separate paginated queries with different per-request caps; chunking - ``skip_geometry`` would emit sub-requests with conflicting + ``skip_geometry`` would emit chunks with conflicting geometry-output settings.""" args = { "monitoring_location_id": ["USGS-A", "USGS-B"], @@ -1317,7 +1317,7 @@ def test_extract_axes_skips_scalar_contract_params(): def test_joint_planner_url_construction_long_filter_and_long_sites(): """Realistic stress: 20 datetime OR-clauses combined with 100 USGS - site IDs. Every sub-request URL built from the plan must fit the + site IDs. Every chunk URL built from the plan must fit the 8000-byte limit, the joint planner must beat the naive "filter at bail-floor, chunk lists" approach, and the partitioned filters must union to the user's original filter expression. @@ -1340,7 +1340,7 @@ def test_joint_planner_url_construction_long_filter_and_long_sites(): filter_expr = " OR ".join(clauses) args = { - "service": "daily", + "collection": "daily", "monitoring_location_id": sites, "filter": filter_expr, } @@ -1349,16 +1349,15 @@ def test_joint_planner_url_construction_long_filter_and_long_sites(): plan = ChunkPlan(args, _construct_api_requests, url_limit) assert plan.total > 1, "expected non-trivial plan for over-limit request" - # Walk every sub-request the plan would issue and assert URL fits. + # Walk every chunk the plan would issue and assert URL fits. over_limit = [] - for sub_args in plan.iter_sub_args(): - req = _construct_api_requests(**sub_args) + for chunk_args in plan.iter_chunk_args(): + req = _construct_api_requests(**chunk_args) url_len = len(str(req.url)) + len(req.content) if url_len > url_limit: - over_limit.append((url_len, sub_args)) + over_limit.append((url_len, chunk_args)) assert not over_limit, ( - f"{len(over_limit)} sub-request(s) exceeded the URL limit; " - f"first: {over_limit[0]}" + f"{len(over_limit)} chunk(s) exceeded the URL limit; first: {over_limit[0]}" ) # Each axis's chunks must union back to its original atoms exactly @@ -1372,9 +1371,7 @@ def test_joint_planner_url_construction_long_filter_and_long_sites(): # Plan must beat the bail-floor-style worst case (singleton sites # × all filter clauses singleton = 500 * 20 = 10,000) — uniform # greedy halving of these inputs cuts that by at least 20×. - assert plan.total < 500, ( - f"joint plan emitted {plan.total} sub-requests (expected <500)" - ) + assert plan.total < 500, f"joint plan emitted {plan.total} chunks (expected <500)" def test_combine_chunk_frames_all_empty_preserves_geo_type(): @@ -1402,13 +1399,13 @@ def test_combine_chunk_frames_single_frame_is_safe_to_mutate(): assert "new_col" not in chunk.columns -def test_iter_sub_args_passthrough_yields_a_copy(): - """``ChunkPlan.iter_sub_args`` yields a fresh dict on every path +def test_iter_chunk_args_passthrough_yields_a_copy(): + """``ChunkPlan.iter_chunk_args`` yields a fresh dict on every path (passthrough and chunked), so a ``fetch_once`` that mutates the dict it receives cannot corrupt ``ChunkPlan.args``.""" args = {"monitoring_location_id": ["USGS-A"], "limit": 100} plan = ChunkPlan(args, _fake_build, url_limit=8000) - sub = next(plan.iter_sub_args()) + sub = next(plan.iter_chunk_args()) sub["monitoring_location_id"] = "mutated" sub["new_key"] = "leaked" assert plan.args["monitoring_location_id"] == ["USGS-A"] @@ -1417,13 +1414,13 @@ def test_iter_sub_args_passthrough_yields_a_copy(): # --- async fan-out path ---------------------------------------------------- # -# Every sub-request is gathered over one ``httpx.AsyncClient`` and +# Every chunk is gathered over one ``httpx.AsyncClient`` and # concurrency is bounded by an ``asyncio.Semaphore`` sized from # ``API_USGS_CONCURRENT`` (the client's connection pool is sized to # match, but the semaphore is the throttle — see ``ChunkedCall._run``). # The conftest's ``_pin_chunker_env`` autouse pins # ``API_USGS_CONCURRENT=1`` (sequential dispatch) for the whole suite; -# each test below raises it so the gather can dispatch sub-requests +# each test below raises it so the gather can dispatch chunks # under a wider cap. The decorated async fetcher is the SAME one used on # both first-run and resume. No real ``httpx.AsyncClient`` round-trip # occurs (the fakes return mock data), even though @@ -1447,7 +1444,7 @@ def _ok_response(remaining=None): return mock.Mock(elapsed=datetime.timedelta(seconds=0.1), headers=headers) -def test_async_fan_out_emits_one_call_per_sub_request(monkeypatch): +def test_async_fan_out_emits_one_call_per_chunk(monkeypatch): """The fan-out hits every sub-args exactly once, dispatched concurrently.""" seen_args = [] @@ -1466,7 +1463,7 @@ async def fetch_async(args): assert sorted({s for tup in seen_args for s in tup}) == sorted( ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10] ) - # Frames concat to one row per sub-request id, in deterministic order. + # Frames concat to one row per chunk id, in deterministic order. assert len(df) == len(seen_args) @@ -1474,7 +1471,7 @@ def test_async_fan_out_aggregates_headers_from_latest_completion(monkeypatch): """Aggregated headers reflect the most recently completed chunk. Completion order can differ from index order in parallel mode, so - rate-limit headers should come from whichever sub-request finished + rate-limit headers should come from whichever chunk finished last, not from the highest sub-args index. """ @@ -1491,23 +1488,23 @@ async def fetch_async(args): def test_async_fan_out_failure_yields_resumable_call(monkeypatch): """A transient 5xx mid-fan-out raises ``ServiceInterrupted`` whose - ``.call`` is a ``ChunkedCall`` holding the completed sub-requests + ``.call`` is a ``ChunkedCall`` holding the completed chunks in a sparse index map. ``exc.call.resume()`` re-issues only the - unfinished sub-requests — through the same async fetcher and the same + unfinished chunks — through the same async fetcher and the same async runner, just on a fresh gather.""" # One async fetcher serves both first-run and resume. On the first - # gather it lets exactly one sub-request succeed and fails the rest + # gather it lets exactly one chunk succeed and fails the rest # transiently; once ``blow_up`` is cleared the resume gather completes - # every still-pending sub-request. ``calls`` counts every invocation + # every still-pending chunk. ``calls`` counts every invocation # across both gathers so we can assert resume only re-issued the owed - # sub-requests. + # chunks. state = {"first_success": False, "blow_up": True} calls = {"n": 0} async def fetch_async(args): calls["n"] += 1 if state["blow_up"]: - # Let the first dispatched sub-request through, fail the rest. + # Let the first dispatched chunk through, fail the rest. if not state["first_success"]: state["first_success"] = True return pd.DataFrame({"id": [_atom_id(args)]}), _ok_response( @@ -1523,11 +1520,11 @@ async def fetch_async(args): interrupted = exc_info.value assert interrupted.call is not None, "interruption must be resumable" - # Exactly one sub-request completed; the rest still owe. + # Exactly one chunk completed; the rest still owe. assert interrupted.completed_chunks == 1 assert interrupted.total_chunks > 1 - # Resume re-issues only the missing sub-requests, via the same async + # Resume re-issues only the missing chunks, via the same async # runner the first run used. state["blow_up"] = False calls_before = calls["n"] @@ -1577,7 +1574,7 @@ async def fetch_async(args): def test_wide_concurrency_uses_async_fetcher_with_no_warning(monkeypatch): """A wide ``API_USGS_CONCURRENT`` is honored directly by the single - async fetcher: every sub-request fans out across it and NO + async fetcher: every chunk fans out across it and NO ``UserWarning`` is emitted.""" calls = [] monkeypatch.setenv("API_USGS_CONCURRENT", "16") @@ -1591,13 +1588,13 @@ async def fetch(args): warnings.simplefilter("error") # any UserWarning would fail the test df, _ = fetch({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) - assert len(calls) > 1 # the gather fanned out across every sub-request + assert len(calls) > 1 # the gather fanned out across every chunk assert len(df) == len(calls) # Eight 20-char sites against ``url_limit=240`` (base 200): any two atoms # joined overflow the 40-byte budget, so the planner lands on eight -# singleton sub-requests — enough fan-out to observe the concurrency gate. +# singleton chunks — enough fan-out to observe the concurrency gate. _EIGHT_SINGLETON_SITES = [f"S{i}" * 10 for i in range(8)] @@ -1626,14 +1623,14 @@ async def fetch_async(args): def test_fan_out_in_flight_high_water_mark_is_the_cap( monkeypatch, cap, expected_high_water ): - """The fetch-level high-water mark of simultaneous sub-requests IS the + """The fetch-level high-water mark of simultaneous chunks IS the ``API_USGS_CONCURRENT`` cap — genuine parallelism up to it, never past - it — and ``unbounded`` degenerates to every sub-request at once. + it — and ``unbounded`` degenerates to every chunk at once. Regression: the cap used to be enforced only by the shared client's - connection-pool size, so sub-requests beyond it queued on connection + connection-pool size, so chunks beyond it queued on connection *acquisition*, subject to the client's pool-acquire timeout (see - ``ChunkedCall._run``). The semaphore parks excess sub-requests before + ``ChunkedCall._run``). The semaphore parks excess chunks before they touch the pool. """ in_flight = {"now": 0, "max": 0} @@ -1643,7 +1640,7 @@ def test_fan_out_in_flight_high_water_mark_is_the_cap( df, _ = fetch({"sites": list(_EIGHT_SINGLETON_SITES)}) - assert len(df) == len(_EIGHT_SINGLETON_SITES) # all sub-requests completed + assert len(df) == len(_EIGHT_SINGLETON_SITES) # all chunks completed assert in_flight["max"] == expected_high_water @@ -1654,15 +1651,15 @@ def test_fan_out_outlives_pool_timeout_on_real_transport(monkeypatch): ``ChunkedCall._run``; at production scale think a batch of large, slowly-streaming pages). - Sub-requests here send real HTTP to a slow localhost server through + Chunks here send real HTTP to a slow localhost server through the chunker's shared client — fakes can't catch this, since ``MockTransport`` bypasses the connection pool. With the pool as the only throttle, 2 connections busy for 0.35 s each and the 0.2 s pool - timeout pinned below, the 2 queued sub-requests sat out the full + timeout pinned below, the 2 queued chunks sat out the full timeout with no completion to reset their clocks → ``httpx.PoolTimeout`` → (retries exhausted, ``API_USGS_RETRIES=0``) a spurious resumable ``ServiceInterrupted``. Gated by the semaphore, - queued sub-requests never touch the pool and the call completes. + queued chunks never touch the pool and the call completes. """ class _SlowHandler(http.server.BaseHTTPRequestHandler): @@ -1695,7 +1692,7 @@ def log_message(self, *args): # keep pytest output clean async def fetch_async(args): client = get_active_client() - assert client is not None, "sub-request must use the shared client" + assert client is not None, "chunk must use the shared client" resp = await client.get(url) assert resp.status_code == 200 return pd.DataFrame({"id": [_atom_id(args)]}), resp @@ -1730,12 +1727,12 @@ async def driver(): # call the sync getter from within a running loop return fetch({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) df, _ = asyncio.run(driver()) - assert len(async_calls) > 1 # every sub-request ran on the async path + assert len(async_calls) > 1 # every chunk ran on the async path assert len(df) == len(async_calls) def test_async_fan_out_cancellation_wins_over_transient_sibling(monkeypatch): - """``asyncio.CancelledError`` raised by any sub-request must + """``asyncio.CancelledError`` raised by any chunk must propagate unmodified, even when a sibling raises a recognized transient (which would otherwise wrap as a resumable :class:`ChunkInterrupted`). Cancellation is asyncio's abort @@ -1993,14 +1990,14 @@ async def afn(): def test_chunker_retries_transient_then_completes(monkeypatch): - """A transient on one sub-request is retried transparently; the + """A transient on one chunk is retried transparently; the decorated call completes with no ChunkInterrupted.""" monkeypatch.setenv("API_USGS_RETRIES", "3") monkeypatch.setattr(_retry_mod.asyncio, "sleep", _aiozero) state = {"failed": False} async def fetch(args): - # Fail the first sub-request once, then succeed everywhere. + # Fail the first chunk once, then succeed everywhere. if not state["failed"]: state["failed"] = True raise RateLimited("429: Too many requests made.") @@ -2042,7 +2039,7 @@ async def fetch(args): sites = list(args["sites"]) if "S1" * 10 in sites: attempts["n"] += 1 - raise ServiceUnavailable("503: service unavailable") + raise ServiceUnavailable("503: collection unavailable") return pd.DataFrame({"sites": sites}), _quota_response(500) decorated = multi_value_chunked(build_request=_fake_build, url_limit=240)(fetch) @@ -2053,7 +2050,7 @@ async def fetch(args): def test_async_fan_out_retries_transient_then_completes(monkeypatch): - """The parallel path retries a transient sub-request and completes.""" + """The parallel path retries a transient chunk and completes.""" monkeypatch.setenv("API_USGS_RETRIES", "3") monkeypatch.setattr(_retry_mod.asyncio, "sleep", _aiozero) @@ -2071,7 +2068,7 @@ async def fetch_async(args): def test_async_fan_out_surfaces_fatal_over_transient(monkeypatch): - """A non-transient bug in one sub-request surfaces raw rather than + """A non-transient bug in one chunk surfaces raw rather than being masked behind a resumable interruption from a transient sibling.""" monkeypatch.setenv("API_USGS_RETRIES", "2") @@ -2142,7 +2139,7 @@ def finalize(frame, response): calls["finalize"] += 1 return frame.assign(finalized=True), ("METADATA", response) - # Fail the 2nd issued sub-request once (the 1st completes, so partial + # Fail the 2nd issued chunk once (the 1st completes, so partial # state is non-empty), then succeed on resume. Conftest pins a single # connection and no retries, so the failure surfaces immediately. state = {"n": 0} @@ -2181,7 +2178,7 @@ async def fetch(args): # passes it through untouched, and any splitting below is the ``n`` cap alone. # ``ChunkPlan`` takes the integer cap (``max_chunks``) directly; # ``parallel_chunks(n)`` publishes ``n`` onto it. The cap bounds the plan's -# *total* sub-request count (the cartesian product across axes), not each axis +# *total* chunk count (the cartesian product across axes), not each axis # independently — see ``test_cap_caps_the_total_across_axes``. # --------------------------------------------------------------------------- @@ -2195,24 +2192,24 @@ def test_default_preserves_passthrough(): plan = ChunkPlan(args, _fake_build, url_limit=8000) # default max_chunks=1 assert plan.axes == [] assert plan.total == 1 - assert list(plan.iter_sub_args()) == [args] + assert list(plan.iter_chunk_args()) == [args] def test_unit_cap_preserves_passthrough(): """``max_chunks=1`` means "no extra fan-out", so a fitting multi-value request stays the trivial passthrough (no axes, ``total == 1``, - ``iter_sub_args`` yields the original args verbatim) — identical to the + ``iter_chunk_args`` yields the original args verbatim) — identical to the default (off), not a materialized one-chunk-per-axis plan.""" args = {"monitoring_location_id": ["A", "B", "C", "D"]} plan = ChunkPlan(args, _fake_build, url_limit=8000, max_chunks=1) assert plan.axes == [] assert plan.total == 1 - assert list(plan.iter_sub_args()) == [args] + assert list(plan.iter_chunk_args()) == [args] @pytest.mark.parametrize("bad", [0, -1]) def test_invalid_cap_raises(bad): - """``max_chunks`` is a sub-request count, so a value below 1 (``0`` or + """``max_chunks`` is a chunk count, so a value below 1 (``0`` or negative) is a caller bug, not a silent no-op: it raises ``ValueError`` at construction. (The public ``parallel_chunks(n)`` already rejects ``n < 1``; this pins the same guard on direct construction.)""" @@ -2249,7 +2246,7 @@ def test_cap_ramps_then_saturates(max_chunks, expected_pieces): def test_cap_bounds_fan_out_for_a_long_axis(): """The cap holds fan-out to ``n``: at ``n=32`` a 100-atom axis fans into ``n`` pieces — NOT 100 singletons — so ``parallel_chunks(32)`` on a huge - list can't detonate into hundreds of sub-requests. Every atom is still + list can't detonate into hundreds of chunks. Every atom is still covered exactly once.""" high = 32 atoms = [f"X{i:03d}" for i in range(100)] @@ -2269,7 +2266,7 @@ def test_cap_below_byte_split_does_not_reduce_fan_out(): A request the byte budget already fans into K>2 chunks is untouched by a cap of 2 (below K), so the byte-driven plan is preserved.""" # Heavy axis of four 30-char atoms; a limit tight enough that the byte pass - # must drive every atom into its own sub-request (4 pieces > the cap of 2). + # must drive every atom into its own chunk (4 pieces > the cap of 2). args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=1) assert baseline.total > 2 # byte pass alone already fanned out past 2 @@ -2280,14 +2277,14 @@ def test_cap_below_byte_split_does_not_reduce_fan_out(): def test_cap_never_exceeds_the_byte_budget(): """Refining on top of an over-budget request keeps the hard invariant: - every sub-request still fits ``url_limit`` (splitting only ever shrinks + every chunk still fits ``url_limit`` (splitting only ever shrinks a chunk), and the fan-out is at least what the byte pass required.""" args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} limit = 310 byte_only = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=1) plan = ChunkPlan(args, _fake_build, url_limit=limit, max_chunks=32) assert plan.total >= byte_only.total - for sub in plan.iter_sub_args(): + for sub in plan.iter_chunk_args(): assert _safe_request_bytes(_fake_build, sub, limit) <= limit @@ -2304,9 +2301,9 @@ def test_cap_refines_the_filter_axis(): def test_cap_caps_the_total_across_axes(): """With more than one multi-value axis the cap bounds the *total* - sub-request count (the cartesian product), not each axis independently — + chunk count (the cartesian product), not each axis independently — the blast-radius guardrail the dial exists for. Two 6-atom axes at a cap - of 4 top out at 4 sub-requests total, not 4x4=16; growth is distributed + of 4 top out at 4 chunks total, not 4x4=16; growth is distributed round-robin across axes rather than one axis alone climbing to the cap.""" args = { "monitoring_location_id": [f"L{i}" for i in range(6)], @@ -2325,7 +2322,7 @@ def test_cap_caps_the_total_across_axes(): def test_cap_bounds_fan_out_across_many_axes(): """The guardrail holds regardless of axis count: three multi-value axes at - a cap of 30 fan out to *at most* 30 sub-requests total — never the + a cap of 30 fan out to *at most* 30 chunks total — never the ``30 ** 3`` a per-axis cap would allow, and never *over* the cap either. 30 is deliberately not evenly reachable by these axes: a single split multiplies the plan by more than one, so the naive ``while total < cap`` @@ -2419,7 +2416,7 @@ def test_parallel_chunks_rejects_non_positive_int(bad): def test_parallel_chunks_drives_end_to_end_fan_out(): """End-to-end: the same fitting request passes through as a single call by - default, but fans into ``n`` sub-requests inside a ``parallel_chunks(n)`` + default, but fans into ``n`` chunks inside a ``parallel_chunks(n)`` block — and the combined result still recovers every atom exactly once.""" sites = [f"S{i:02d}" for i in range(8)] @@ -2439,7 +2436,7 @@ async def fetch(args): calls.clear() with parallel_chunks(8): df_fine, _ = fetch({"monitoring_location_id": sites}) - # 8 atoms at n=8 → 8 singleton sub-requests. + # 8 atoms at n=8 → 8 singleton chunks. assert len(calls) == 8 assert all(len(chunk) == 1 for chunk in calls) # Union across chunks recovers the original set, once each. @@ -2450,7 +2447,7 @@ async def fetch(args): @pytest.mark.parametrize("n", [1, 2, 3, 8]) def test_parallel_chunks_supports_arbitrary_n(n): """An arbitrary ``n`` (not only 2/8/32) fans an under-limit request into - exactly ``n`` sub-requests, together covering every site once — including + exactly ``n`` chunks, together covering every site once — including ``n=1``, the explicit no-op that stays a single passthrough call.""" sites = [f"S{i:02d}" for i in range(8)] calls: list[int] = [] diff --git a/tests/waterdata_filters_test.py b/tests/waterdata_filters_test.py index 207a1dcf..8406c0db 100644 --- a/tests/waterdata_filters_test.py +++ b/tests/waterdata_filters_test.py @@ -44,7 +44,7 @@ def test_quote_cql_str_doubles_embedded_quotes(): def test_construct_filter_lang_hyphenated(): """The Python kwarg `filter_lang` is sent as URL key `filter-lang`.""" req = _construct_api_requests( - service="continuous", + collection="continuous", monitoring_location_id="USGS-07374525", parameter_code="72255", filter="time >= '2023-01-01T00:00:00Z'", @@ -99,7 +99,7 @@ def test_split_top_level_or_single_clause(): @pytest.mark.parametrize( - "service", + "collection", [ "daily", "continuous", @@ -111,10 +111,10 @@ def test_split_top_level_or_single_clause(): "channel-measurements", ], ) -def test_construct_filter_on_all_ogc_services(service): +def test_construct_filter_on_all_ogc_services(collection): """Filter passthrough works uniformly for every OGC collection endpoint.""" req = _construct_api_requests( - service=service, + collection=collection, filter="value > 0", filter_lang="cql-text", ) @@ -143,9 +143,9 @@ def _filter_size_aware_build(**kwargs): def test_long_filter_fans_out_into_multiple_requests(): """An oversized top-level OR filter triggers multiple HTTP - sub-requests via the joint planner; every original clause is - preserved across sub-requests; results concatenate to one row per - sub-request given the one-row-per-chunk mock.""" + chunks via the joint planner; every original clause is + preserved across chunks; results concatenate to one row per + chunk given the one-row-per-chunk mock.""" expr = _filter_chunking_clauses() sent_filters: list[str] = [] @@ -181,7 +181,7 @@ async def fake_walk_pages(*, geopd, req): def test_long_filter_deduplicates_cross_chunk_overlap(): - """Features returned by multiple sub-requests with the same ``id`` + """Features returned by multiple chunks with the same ``id`` are deduplicated in the concatenated result.""" expr = _filter_chunking_clauses() call_count = {"n": 0} @@ -215,7 +215,7 @@ async def fake_walk_pages(*_args, **_kwargs): def test_empty_chunks_do_not_downgrade_geodataframe(): - """A mix of empty and non-empty sub-request responses must not + """A mix of empty and non-empty chunk responses must not downgrade a GeoDataFrame-typed result to a plain DataFrame. ``_get_resp_data`` returns ``pd.DataFrame()`` on empty responses, which would otherwise strip geometry/CRS from the concatenated diff --git a/tests/waterdata_progress_test.py b/tests/waterdata_progress_test.py index de5618f3..2f789f43 100644 --- a/tests/waterdata_progress_test.py +++ b/tests/waterdata_progress_test.py @@ -411,7 +411,7 @@ def test_walk_pages_reports_pages_and_rate_limit(): assert len(df) == 2 out = stream.getvalue() - # The service set on the context reaches _paginate's render via the contextvar. + # The collection set on the context reaches _paginate's render via the contextvar. assert "Retrieving: daily ·" in out assert "2 pages" in out assert "4,998 requests remaining" in out @@ -521,8 +521,8 @@ async def run(): def test_fan_out_async_sets_chunks_on_active_reporter(monkeypatch): """The async fan-out core (``ChunkedCall._run``) records ``plan.total`` on the active reporter so the progress line knows how - many sub-requests are in flight, and ticks ``current_chunk`` via - ``start_chunk(len(completed))`` as each gathered sub-request finishes + many chunks are in flight, and ticks ``current_chunk`` via + ``start_chunk(len(completed))`` as each gathered chunk finishes — reaching ``plan.total`` in the all-success case.""" # Fake build_request whose URL length scales with the sites list, @@ -559,7 +559,7 @@ async def run(): total_recorded, current_recorded = asyncio.run(run()) assert total_recorded == plan.total - # Each sub-request that completes bumps current_chunk via + # Each chunk that completes bumps current_chunk via # start_chunk(len(completed)), so by the time the gather finishes # current_chunk reflects the total number of successful chunks — # plan.total in the all-success case. diff --git a/tests/waterdata_queryables_test.py b/tests/waterdata_queryables_test.py index 2ac7d5db..9494c18a 100644 --- a/tests/waterdata_queryables_test.py +++ b/tests/waterdata_queryables_test.py @@ -128,7 +128,7 @@ def _items_query(httpx_mock): def test_passthrough_queryables_sent_as_filters(httpx_mock): """An OGC getter forwards queryables that aren't in its explicit signature - (e.g. ``state_name``, ``site_type_code``) to the service as query filters, + (e.g. ``state_name``, ``site_type_code``) to the collection as query filters, alongside the named params.""" _mock_daily(httpx_mock) diff --git a/tests/waterdata_test.py b/tests/waterdata_test.py index 5c54c9fe..ae9b908a 100644 --- a/tests/waterdata_test.py +++ b/tests/waterdata_test.py @@ -2,6 +2,7 @@ import datetime import json import re +import warnings from pathlib import Path from unittest import mock from urllib.parse import parse_qs, urlsplit @@ -48,7 +49,7 @@ _OGC_BASE = "https://api.waterdata.usgs.gov/ogcapi/v0" _STATS_BASE = "https://api.waterdata.usgs.gov/statistics/v0" -#: Two real features per collection, captured from the live service and trimmed. +#: Two real features per collection, captured from the live collection and trimmed. #: Property names, nesting, and value types (including the numeric-looking #: strings the API really sends) are verbatim; only the row count is reduced. #: Regenerate a collection by re-querying it with ``limit=2`` and replacing that @@ -63,7 +64,7 @@ def _activate_waterdata_dialect(): """Make the Water Data OGC base URL and dialect ambient for this module. Both are normally set together by ``get_ogc_data`` per call: the base URL - (the OGC package names no service of its own) and the dialect + (the OGC package names no collection of its own) and the dialect (monitoring-locations -> POST/CQL2; daily -> date-only time args). The direct ``_construct_api_requests``/``_construct_cql_request`` unit tests here bypass that entry point, so activate both module-wide so they @@ -323,7 +324,7 @@ def test_check_profiles(): def test_construct_api_requests_multivalue_get(): - """Multi-value params use GET with comma-separated values for daily service.""" + """Multi-value params use GET with comma-separated values for daily collection.""" req = _construct_api_requests( "daily", monitoring_location_id=["USGS-05427718", "USGS-05427719"], @@ -356,7 +357,7 @@ def test_construct_api_requests_monitoring_locations_post(): # Body is serialized compactly (tight separators, no whitespace): the # body counts against the server's ~8 KB request-size cap and the # chunk planner's byte budget, so pretty-printing would needlessly - # halve how many ids fit per sub-request and double the chunk count. + # halve how many ids fit per chunk and double the chunk count. raw = req.content.decode() assert "\n" not in raw and ", " not in raw and ": " not in raw @@ -408,23 +409,46 @@ def test_construct_cql_request_skip_geometry_none_omits_param(): assert "skipGeometry" not in str(req.url) +def test_get_cql_service_keyword_is_deprecated_but_works(): + """``service=`` still resolves to ``collection`` for one deprecation window. + + ``service`` was the published spelling, and OGC API - Features calls the + value a collection -- it is the ``collectionId`` in ``/collections/{id}``. + The rename must not silently change behavior for callers using the old name. + """ + with pytest.warns(DeprecationWarning, match="use 'collection'"): + with pytest.raises(ValueError, match="Unknown collection"): + get_cql(service="not-a-collection", cql="a=1") + + # The new spelling emits nothing. + with warnings.catch_warnings(): + warnings.simplefilter("error", DeprecationWarning) + with pytest.raises(ValueError, match="Unknown collection"): + get_cql(collection="not-a-collection", cql="a=1") + + # Passing both spellings is ambiguous and refused, which the hand-rolled + # shim this replaced did not do -- it silently dropped ``service``. + with pytest.raises(TypeError, match="received both"): + get_cql(service="daily", collection="daily", cql="a=1") + + def test_get_cql_unknown_service_raises(): - """An unknown service is rejected before any network call.""" - with pytest.raises(ValueError, match="Unknown service"): - get_cql("not-a-service", {"op": "isNull", "args": [{"property": "x"}]}) + """An unknown collection is rejected before any network call.""" + with pytest.raises(ValueError, match="Unknown collection"): + get_cql("not-a-collection", {"op": "isNull", "args": [{"property": "x"}]}) def test_waterdata_services_literal_matches_output_id_map(): - """The WATERDATA_SERVICES Literal and _OUTPUT_ID_BY_SERVICE must enumerate - the same services: get_cql validates against the dict while the Literal - types the public signature, so drift would let one accept a service the other + """The WATERDATA_SERVICES Literal and _OUTPUT_ID_BY_COLLECTION must enumerate + the same collections: get_cql validates against the dict while the Literal + types the public signature, so drift would let one accept a collection the other rejects.""" from typing import get_args from dataretrieval.waterdata.types import WATERDATA_SERVICES - from dataretrieval.waterdata.utils import _OUTPUT_ID_BY_SERVICE + from dataretrieval.waterdata.utils import _OUTPUT_ID_BY_COLLECTION - assert set(get_args(WATERDATA_SERVICES)) == set(_OUTPUT_ID_BY_SERVICE) + assert set(get_args(WATERDATA_SERVICES)) == set(_OUTPUT_ID_BY_COLLECTION) def test_construct_api_requests_single_value_stays_get(): @@ -492,7 +516,7 @@ def test_construct_api_requests_two_element_date_list_becomes_interval(): # --- mocked getter smoke tests ------------------------------------------------ # These replace what used to be ~34 live calls to the Water Data API. Each one # serves a committed fixture (``tests/data/waterdata_ogc_fixtures.json``, two -# real features per collection captured from the service) and asserts what we +# real features per collection captured from the collection) and asserts what we # actually control: that the request we build carries the right params, and that # the frame we hand back has the right columns, dtypes, and ordering. # @@ -548,7 +572,7 @@ def _sent(httpx_mock, collection=None): # --- samples (CSV) ----------------------------------------------------------- -# The samples service is CSV over a different host, so these use small CSV +# The samples collection is CSV over a different host, so these use small CSV # bodies rather than the GeoJSON fixtures. _SAMPLES_RE = re.compile(r"^https://api\.waterdata\.usgs\.gov/samples-data/") @@ -581,7 +605,7 @@ def test_samples_results(httpx_mock): @pytest.mark.parametrize( - ("service", "profile", "kwargs", "expect_path"), + ("collection", "profile", "kwargs", "expect_path"), [ ( "activities", @@ -610,12 +634,12 @@ def test_samples_results(httpx_mock): ], ) def test_samples_service_profile_routes_to_its_endpoint( - httpx_mock, service, profile, kwargs, expect_path + httpx_mock, collection, profile, kwargs, expect_path ): - """Each ``service``/``profile`` pair addresses ``//``. + """Each ``collection``/``profile`` pair addresses ``//``. - Previously one live test per service asserted a column count against real - data (``len(df.columns) == 97``), which broke whenever the service added a + Previously one live test per collection asserted a column count against real + data (``len(df.columns) == 97``), which broke whenever the collection added a field. What is ours to get right is the routing and the parse, so that is what this checks. """ @@ -625,7 +649,7 @@ def test_samples_service_profile_routes_to_its_endpoint( text="Org_Identifier,Location_Identifier\nUSGS-WI,USGS-06719505\n", ) - df, _ = get_samples(service=service, profile=profile, **kwargs) + df, _ = get_samples(service=collection, profile=profile, **kwargs) url = str(httpx_mock.get_requests()[0].url) assert expect_path in url @@ -636,7 +660,7 @@ def test_samples_service_profile_routes_to_its_endpoint( def test_get_daily(httpx_mock): - """A daily query returns tidy rows with the service id renamed to + """A daily query returns tidy rows with the collection id renamed to ``daily_id`` and moved last, dates as ``date`` objects, values numeric.""" _mock_items(httpx_mock, "daily") @@ -673,7 +697,7 @@ def test_get_daily_sends_date_only_time_interval(httpx_mock): def test_get_daily_properties(httpx_mock): """``properties`` selects and orders the output columns, and is forwarded to - the service so it does the projection too.""" + the collection so it does the projection too.""" requested = [ "daily_id", "monitoring_location_id", @@ -695,14 +719,14 @@ def test_get_daily_properties(httpx_mock): assert df.shape[1] == len(requested) # ``daily_id`` is our name for the wire's ``id`` and ``geometry`` is governed # by ``skipGeometry``, not by ``properties`` -- neither is a real queryable, - # so neither may be forwarded or the service would reject the projection. + # so neither may be forwarded or the collection would reject the projection. sent = _sent(httpx_mock, "daily")[0]["properties"][0].split(",") assert "daily_id" not in sent and "geometry" not in sent assert sent == ["monitoring_location_id", "parameter_code", "time", "value"] def test_get_daily_properties_id(httpx_mock): - """``'id'`` in ``properties`` resolves to the service-specific id column + """``'id'`` in ``properties`` resolves to the collection-specific id column while keeping the caller's requested position.""" _mock_items(httpx_mock, "daily") @@ -786,7 +810,7 @@ def test_get_latest_daily(httpx_mock): def test_get_latest_daily_properties_geometry(httpx_mock): """Geometry survives an explicit ``properties`` list that omits it -- the - service returns it regardless unless ``skip_geometry`` is set, so the + collection returns it regardless unless ``skip_geometry`` is set, so the projection must not drop it.""" _mock_items(httpx_mock, "latest-daily") @@ -884,7 +908,7 @@ def test_get_cql_str_body_sent_verbatim(httpx_mock): def test_get_cql_properties_id_translation(httpx_mock): - """``properties=['id', ...]`` resolves ``id`` to the service's output id + """``properties=['id', ...]`` resolves ``id`` to the collection's output id column, preserving the requested order.""" cql = { "op": "in", @@ -951,7 +975,7 @@ def test_get_field_measurements_metadata(httpx_mock): def test_get_field_measurements_metadata_multi_site(httpx_mock): - """Multiple sites plus a parameter filter reach the service in one + """Multiple sites plus a parameter filter reach the collection in one request.""" sites = ["USGS-07069000", "USGS-07064000", "USGS-07068000"] _mock_items(httpx_mock, "field-measurements-metadata") @@ -1147,11 +1171,11 @@ def test_get_reference_table_accepts_numpy_int_max_rows(httpx_mock): # values); these pin the flattening, which is the part we own. -def _mock_stats(httpx_mock, service): +def _mock_stats(httpx_mock, collection): httpx_mock.add_response( method="GET", - url=re.compile(rf"^{re.escape(_STATS_BASE)}/{service}"), - json=_fixture(service), + url=re.compile(rf"^{re.escape(_STATS_BASE)}/{collection}"), + json=_fixture(collection), ) diff --git a/tests/waterdata_utils_test.py b/tests/waterdata_utils_test.py index 0d7dd6ba..e805ec92 100644 --- a/tests/waterdata_utils_test.py +++ b/tests/waterdata_utils_test.py @@ -71,7 +71,7 @@ def _run_walk_pages(*, geopd, req, client): def test_get_args_basic(): local_vars = { "monitoring_location_id": "USGS-123", - "service": "daily", + "collection": "daily", "output_id": "daily_id", "none_val": None, "other": "val", @@ -83,7 +83,7 @@ def test_get_args_basic(): def test_get_args_with_exclude(): local_vars = { "monitoring_location_id": "USGS-123", - "service": "daily", + "collection": "daily", "output_id": "daily_id", "to_exclude": "secret", "other": "val", @@ -210,7 +210,7 @@ def _page(idx, *, has_next): def test_finalize_ogc_truncates_combined_to_max_rows(): # max_rows is enforced on the *combined* frame in _finalize_ogc (after # dedup/sort), so it bounds the total exactly even when a chunked call's - # per-sub-request pages overshoot the per-_paginate early-stop. + # per-chunk pages overshoot the per-_paginate early-stop. frame = pd.DataFrame({"id": [str(i) for i in range(10)]}) resp = mock.MagicMock() resp.url = "https://example.com/q" @@ -223,7 +223,7 @@ def test_finalize_ogc_truncates_combined_to_max_rows(): properties=None, output_id="thing_id", convert_type=False, - service="things", + collection="things", max_rows=3, ) assert len(df) == 3 @@ -462,7 +462,7 @@ def _stats_initial_ok(): "features": [], } resp.headers = {} - resp.url = "https://example.com/stats?service=foo" + resp.url = "https://example.com/stats?collection=foo" return resp @@ -689,7 +689,7 @@ def test_handle_nesting_tolerates_missing_features_key(): def test_get_resp_data_always_materializes_id_column(): """``_get_resp_data`` must always materialize the ``id`` column (NaN-filled when no feature carries one) so the downstream - ``_arrange_cols`` rename to the service-specific output_id + ``_arrange_cols`` rename to the collection-specific output_id (``daily_id``, ``channel_measurements_id``, etc.) isn't a silent no-op.""" resp = mock.MagicMock() @@ -1120,8 +1120,8 @@ def test_ogc_getter_resolves_state_at_getter_layer(monkeypatch): captured: dict = {} - def fake_get_ogc_data(args, service, *a, **k): - captured.update(args=args, service=service) + def fake_get_ogc_data(args, collection, *a, **k): + captured.update(args=args, collection=collection) return pd.DataFrame(), mock.Mock() monkeypatch.setattr(_metadata, "get_ogc_data", fake_get_ogc_data) @@ -1135,7 +1135,7 @@ def test_get_ogc_data_wrapper_does_not_touch_state(): query dict (e.g. from ``get_reference_table``) is forwarded untouched.""" captured: dict = {} - def fake_engine_get_ogc_data(args, service, output_id, **k): + def fake_engine_get_ogc_data(args, collection, output_id, **k): captured["args"] = dict(args) return pd.DataFrame(), mock.Mock()