Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7669847
fix(review-gateway): require tool evidence for free review requests
seonghobae Sep 24, 2026
a88a4de
fix(review-gateway): stop ambiguous free completion replay
seonghobae Sep 24, 2026
b63cb48
fix: keep review replay decisions candidate local
seonghobae Sep 24, 2026
081ae11
experiment: compose #1209 and #1227 review gates
seonghobae Sep 25, 2026
0142f61
fix(route): advance free candidates after explicit 429
seonghobae Sep 25, 2026
3853908
Merge commit '84736f4d9840b3d06edb8fdcc7210abddd86175e' into probe/co…
seonghobae Sep 25, 2026
0cf0a3c
test(route): cover 429 storm with default retry budget
seonghobae Sep 25, 2026
6713563
experiment: expose review 429 failover gap
seonghobae Sep 25, 2026
23339e4
Merge commit '0142f6129c22880b680d84b01d81c37f7cb060d6' into probe/co…
seonghobae Sep 25, 2026
fa045f4
experiment: cover review quota rejection through HTTP and proxy
seonghobae Sep 25, 2026
7ff1a65
experiment: recover review routes after explicit quota rejection
seonghobae Sep 25, 2026
a8f839f
Merge commit 'aa00d635b48589a26b705a37506ef2923549956d' into probe/co…
seonghobae Sep 25, 2026
0e45bc4
experiment: expose structured review quota gap
seonghobae Sep 25, 2026
8e3496d
experiment: preserve bounded review 429 receipts
seonghobae Sep 25, 2026
d873f6e
experiment: guard wrapped quota cause from review replay
seonghobae Sep 25, 2026
9625700
experiment: require direct quota rejection for review replay
seonghobae Sep 25, 2026
fc48731
experiment: keep wrapped provider failures sticky for reviews
seonghobae Sep 25, 2026
85deff4
Merge commit '0cf0a3cbe77af4171a333262f10974a3a0f3b31e' into probe/co…
seonghobae Sep 25, 2026
038869c
docs(review): distinguish explicit quota rejection from unknown send
seonghobae Sep 25, 2026
c1c07ce
merge: preserve free review routing and conduct error receipts
seonghobae Sep 27, 2026
4e80201
merge: preserve review tool evidence across image-aware free routing
seonghobae Sep 27, 2026
147394d
Merge remote-tracking branch 'origin/main' into prep/1227-main-integr…
seonghobae Sep 28, 2026
0b4d250
test(review): close synthetic provider errors after assertions
seonghobae Sep 28, 2026
605b6e3
fix: preserve zero-wait structured failover
seonghobae Oct 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
retries only final synthesis on a ready route. After HTTP 413 retires one
route, another rate-limited route can still recover. This repair changes
retry timing, not the Chat Completions JSON-schema request or response
fields; explicit model pins and expired budgets remain fail-closed.
fields; explicit model pins and expired budgets remain fail-closed. A zero
wait budget now limits only an all-cooling pool and does not prevent
immediate failover from an explicit 429 to an already-ready sibling.
- Structured non-stream `/v1/responses` requests now preflight every required
conduct role for `orchestrator/free`. Image-capable worker-only pools fail
closed with typed HTTP 400 before template planning instead of surfacing a
Expand Down
247 changes: 187 additions & 60 deletions contextual_orchestrator/orchestrator.py

Large diffs are not rendered by default.

11 changes: 11 additions & 0 deletions contextual_orchestrator/provider_errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,17 @@ def classify_provider_failure(
# holding just the classified ProviderUpstreamError -- e.g. _invoke's
# chat transport -- can still record and wait out the same cooldown.
extra_detail: dict[str, Any] = {}
if status in (404, 410):
try:
body = _json.loads(provider_error_body(exc))
except Exception: # noqa: BLE001 - provider error bodies are untrusted streams
body = None
if (
isinstance(body, dict)
and isinstance(body.get("error"), dict)
and body["error"].get("code") == "model_not_found"
):
extra_detail["model_refusal_proven"] = True
if status in (429, 503):
retry_after = resolve_retry_after_seconds(exc)
if retry_after is not None:
Expand Down
14 changes: 6 additions & 8 deletions contextual_orchestrator/review_gateway.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,10 @@

@dataclass(frozen=True)
class ReviewModelAdmission:
"""Typed, request-scoped provenance for one admitted review-pool model.
"""Typed catalog provenance for one admitted review-pool model.

Every field is owner-produced evidence about *why* the candidate is
admitted, so a consumer never has to re-derive eligibility, re-probe
readiness, or apply its own provider/model/fallback heuristics.
These fields explain catalog eligibility. They do not certify that a
particular request shape is supported or that the provider is live.
"""

contract_version: str
Expand Down Expand Up @@ -118,11 +117,10 @@ def review_model_admission(
def review_pool_admissions(
agents: Sequence[Any],
) -> list[ReviewModelAdmission]:
"""Project a built review pool into versioned, consumer-readable provenance.
"""Project a built review pool into versioned catalog provenance.

The owner exposes this so a caller can send only the gateway token and
``model: orchestrator/free`` -- no provider/model/fallback parameters,
no credential eligibility, no candidate catalog, no probing.
The caller sends only the gateway token and ``model: orchestrator/free``;
request-specific admission and execution remain gateway responsibilities.
"""
return [
ReviewModelAdmission(
Expand Down
6 changes: 6 additions & 0 deletions docs/doctoring/autonomous_kpi_runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,12 @@ owner tests and protected release. See [the evidence receipt](https://github.com
These are failed deliveries in the accuracy denominator, not measured routing
decision latencies. Their elapsed times include work beyond initial selection.

## Noema free-pool 429 follow-up, 2026-09-25

Required Noema Review run [36026163711](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/36026163711), job `107824690321`, used sidecar source `767e67fbc6b881a452761f32abb69b9971b9b03b` for PR #1231. Its `noema-sidecar-evidence` artifact `10844122590` records 19 provider attempts across eight distinct candidates for one request ID. After several NIM attempts, two different OpenRouter free models each explicitly returned 429. The 1197.3-second caller attempt then ended with 429. Thus this run proves cross-candidate routing occurred; it does not prove that another eligible, ready candidate remained after the final rejection. The preflight's six ready routes are earlier liveness evidence, not the request's terminal candidate count.

The Noema workflow scheduled a bounded same-head continuation after 131 seconds, but its `POST /repos/{repo}/dispatches` failed with `Resource not accessible by integration` (403). This is a separate workflow credential/permission blocker. The current gateway code also retried an explicitly 429-rejected candidate before advancing when `tool_retry_attempts` was nonzero; the tool-bearing `orchestrator/free` HTTP regression for this repair asserts one call to that candidate followed by one call to the next eligible candidate. It does not authorize replay of ambiguous transport failures or claim that the pinned sidecar contains this repair. Recheck the exact sidecar source, hosted request trace, and current-head review verdict before release or attribution of a future 429.

### Structured synthesis 429 follow-up, 2026-09-25

[Noema run 36024200990](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/runs/36024200990)
Expand Down
40 changes: 40 additions & 0 deletions docs/product-technical-gap-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,46 @@ environment's missing locked `pytest-asyncio` and inherited deprecated
`jsonschema.RefResolver` warnings remain explicit. Hosted exact-head gates,
independent review, protected merge, immutable release, and consumer adoption
remain required.
## 2026-09-25 review tool-request admission (#1106, proposed)

The #940 baseline below intentionally allowed a free model with unknown tool
support to receive a tool request. Issue #1106's later Strix tool-call 404
shows why plain-chat readiness cannot authorize that request shape. The owner
now requires positive discovery evidence for review-pool tool calls: the
existing `tool_call:single|multi` tags determine admission for the actual
request, and an empty eligible set returns typed 503 before provider send.
Focused RED reproduced unknown-evidence admission, missing fail-closed
behavior, and replay after ambiguous failures. The locked local environment
passed 388 neighboring tests on 2026-09-25. Four provider-reliability tests
were deselected after reproducing the same failures on unmodified `origin/main`
at `5665b0ad` (selection-design receipt and provider allowlist classification).
This is local contract evidence, not live provider readiness, judged review
quality, protected delivery, or issue #1106 completion. Calibrated allocation,
immutable release, and the central consumer's preflight removal remain open.
The follow-up RED showed unsafe review replay after a post-send timeout;
review-tagged completions now stop on that unknown outcome. A separate RED
showed that the review route stopped after an explicit provider 429 even with
another eligible free candidate. The owner now records the rejected candidate's
cooldown and advances on direct 429, including HTTP tool requests and JSON-schema
synthesis. An all-429 pool waits only within its configured budget and otherwise
returns typed 429. A wrapped error with a nested 429 remains sticky because the
outer send's outcome is unknown. Direct pre-send local-slot failure can still
advance; HTTP 503 and post-send timeout remain terminal. Synthetic transport
call counts and typed errors cover these boundaries. This source evidence does
not establish provider idempotency, a protected release, or Noema approval;
the observed Noema job used the older `767e67fb` sidecar pin.

Exact-parent follow-up `0b4d2503f1f72aba4dc0cd05ce5f1504425b3e36`
found that structured synthesis created a zero-duration recovery deadline
after the first explicit 429, then enforced it before checking whether another
eligible candidate was already ready. The RED case called only the primary and
raised `rate_limited_storm`; the same test already proved ordinary-budget 429
advancement and ambiguous-503 no-replay. GREEN scopes deadline expiry to the
existing all-candidates-cooling branch, so `rate_limit_wait_seconds=0` still
permits immediate ready-sibling failover while an all-429 pool returns typed
429 without replay. The two adjacent owner suites pass 178 tests. This remains
Proposed pending exact-head hosted checks, independent approval, protected
integration, immutable release, and consumer adoption.

## 2026-09-08 item-covariate two-group boundary repair (proposed)

Expand Down
45 changes: 45 additions & 0 deletions docs/research/review-gateway-free-pool-admission.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,51 @@ No new routing heuristic is introduced by this change.

## Executable provenance

### Request-shaped tool admission (2026-09-25, proposed)

For a review-pool `orchestrator/free` Chat Completions request with tools, the
gateway now uses its existing discovery evidence before sending to a provider.
`tool_call:multi` admits either one or several tools; `tool_call:single` admits
a request that does not require parallel calls. A review candidate without
either positive signal can still serve plain chat, but cannot serve a tool
request. If no free candidate proves the requested tool shape, the gateway
returns `503 request_capability_unavailable` with `capability=tool_call` and
does not send a completion. The same eligible set reaches every conduct role
and final synthesis when the caller requests an orchestrated response. This
addresses the plain-chat-ready/tool-404
counterexample in issue #1106 without a provider-name branch in the consumer.

The signal proves only the probed tool-call contract. It does not prove live
availability, output quality, a provider idempotency guarantee, or completion
of a long review. The catalog admission result remains distinct from an
execution receipt. Calibrated allocation, a released request-scoped contract,
immutable owner release, and the consumer's deletion of its own preflight
remain open acceptance conditions for issue #1106.

### Completion replay boundary

For a review-tagged `orchestrator/free` candidate, a read timeout or connection
reset after the transport began has unknown outcome and stops the request with
`provider_outcome_unknown`; a second provider is not called. An explicit HTTP
429 is a quota rejection: the gateway records its cooldown and may try another
eligible candidate, or wait within its bounded budget when every candidate is
cooling. A wrapped failure that merely contains a 429 does not establish that
the outer send was rejected. HTTP 503, 408, 409, and 425 remain terminal for
the review request. Direct local-slot admission failure
before the transport call, or a provider response explicitly identifying
`model_not_found` or a request-size rejection,
can advance to another eligible candidate. The test counts transport calls
on each side of the local slot and asserts that an unknown outcome never
causes a second send. This narrows the earlier virtual-selector failover
behavior for the review pool; other virtual selectors retain their prior
contract. The same no-replay rule follows each actual review-tagged candidate
through route and conduct calls for uncertain outcomes, including a mixed free
pool. A non-review
candidate retains its prior retry policy. A bodyless HTTP 404 cannot authorize
another review send. No provider idempotency agreement has been established.
This matches the non-idempotent retry boundary in
[RFC 9110 §9.2.2](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2).

The PR implementing this contract must prove at least the following cases:

- all five required provider credentials may be supplied and registered together;
Expand Down
Loading
Loading