Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ba0f90e
feat: add free-first fallback policy facade
seonghobae Aug 4, 2026
9207507
feat: add validated fallback policy value objects
seonghobae Aug 4, 2026
8295f3b
feat: add deterministic fallback planner
seonghobae Aug 4, 2026
f71790c
feat: add strict fallback manifest parser
seonghobae Aug 4, 2026
032db15
feat: add workflow fallback policy CLI
seonghobae Aug 4, 2026
3cb2e8a
feat: export fallback policy API
seonghobae Aug 4, 2026
c04239f
test: make fallback test helpers importable
seonghobae Aug 4, 2026
85a53af
test: add fallback policy fixtures
seonghobae Aug 4, 2026
4bb2e39
test: cover free-first planning and validation
seonghobae Aug 4, 2026
ae0c8ea
test: cover strict fallback manifests
seonghobae Aug 4, 2026
7570698
test: cover fallback policy CLI and secret boundaries
seonghobae Aug 4, 2026
5a01d4e
docs: document fallback policy integration
seonghobae Aug 4, 2026
87cc85f
docs: add fallback policy doctoring record
seonghobae Aug 4, 2026
3626700
docs: record free-first fallback policy
seonghobae Aug 4, 2026
82ea37e
Merge security transport base into free-first fallback policy
seonghobae Aug 4, 2026
b93d161
docs: reconcile fallback and portable-lock changelog
seonghobae Aug 5, 2026
40c6a4b
Merge b93d1615855883cad052ac7cdf32a6cd979e0d82 into cfd42f309ea39a189…
seonghobae Aug 5, 2026
0f76a38
test(fallback): prohibit environment secret inspection
seonghobae Aug 5, 2026
5fa5dba
ci(fallback): execute the credential-boundary quality gate
seonghobae Aug 5, 2026
11cb125
ci(fallback): verify stacked feature heads on push
seonghobae Aug 5, 2026
a1799db
test(fallback): isolate provider credential reads
seonghobae Aug 5, 2026
8e2c586
fix(fallback): accept declared credential names
seonghobae Aug 5, 2026
dad01e8
docs(fallback): document credential-name boundary
seonghobae Aug 5, 2026
e3b814f
docs(changelog): record fallback secret boundary
seonghobae Aug 5, 2026
0b04029
docs(fallback): record the credential-name trust boundary
seonghobae Aug 5, 2026
e225934
test(fallback): reject manifest scalar type confusion
seonghobae Aug 7, 2026
344f478
fix(fallback): validate identifier scalar types before regex
seonghobae Aug 7, 2026
fde76c0
fix(fallback): reject manifest type confusion
seonghobae Aug 7, 2026
3e6afc4
docs(changelog): record fallback manifest type hardening
seonghobae Aug 7, 2026
a369e43
test(fallback): reject mutable credential controls
seonghobae Aug 7, 2026
06f0010
fix(fallback): freeze credential control collections
seonghobae Aug 7, 2026
c995310
docs(changelog): record immutable fallback controls
seonghobae Aug 7, 2026
49a6d15
fix(fallback): preserve credential validation diagnostics
seonghobae Aug 7, 2026
5104ea1
fix(fallback): keep credential branch coverage fail closed
seonghobae Aug 7, 2026
73ed3a0
merge: refresh fallback policy onto security head
seonghobae Aug 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/workflows/model-fallback-policy-verify.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Model fallback policy quality

on:
pull_request:
paths:
- contextual_orchestrator/_fallback_*.py
- contextual_orchestrator/model_fallback.py
- tests/fallback_test_support.py
- tests/test_model_fallback_*.py
- docs/model-fallback-policy.md
- docs/doctoring/free-first-model-fallback.md
- .github/workflows/model-fallback-policy-verify.yml
push:
paths:
- contextual_orchestrator/_fallback_*.py
- contextual_orchestrator/model_fallback.py
- tests/fallback_test_support.py
- tests/test_model_fallback_*.py
- docs/model-fallback-policy.md
- docs/doctoring/free-first-model-fallback.md
- .github/workflows/model-fallback-policy-verify.yml

permissions:
contents: read

concurrency:
group: model-fallback-policy-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
verify:
name: Fallback statement, branch, docstring, and secret-boundary evidence
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Harden the runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
with:
egress-policy: audit

- name: Checkout exact source revision without credentials
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"

- name: Install hash-locked verification dependencies
run: |
python -m pip install --require-hashes -r fuzz/requirements-property.txt
python -m pip install --require-hashes -r requirements-opencode-review-ci.txt

- name: Run fallback behavior and quality gates
run: |
set -euo pipefail
python -m compileall -q contextual_orchestrator tests
python -m pytest -q \
tests/test_model_fallback_cli.py \
tests/test_model_fallback_manifest.py \
tests/test_model_fallback_plan.py
python -m coverage erase
python -m coverage run --branch --source=contextual_orchestrator -m pytest -q \
tests/test_model_fallback_cli.py \
tests/test_model_fallback_manifest.py \
tests/test_model_fallback_plan.py
python -m coverage report \
--include='contextual_orchestrator/_fallback_*.py,contextual_orchestrator/model_fallback.py' \
--show-missing \
--fail-under=100
python -m interrogate -f 100 \
contextual_orchestrator/_fallback_cli.py \
contextual_orchestrator/_fallback_manifest.py \
contextual_orchestrator/_fallback_plan.py \
contextual_orchestrator/_fallback_types.py \
contextual_orchestrator/model_fallback.py
git diff --check
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and

## [Unreleased]

### Added

- Add a transport-neutral, versioned model fallback policy that validates explicit cost tiers and deterministically exhausts eligible free candidates before any paid fallback.
- Filter fallback candidates by repository visibility, required capability, and configured credential name without retaining or serializing secret values.
- Add a standard-library CLI for immutable cross-repository workflow integration, with complete statement, branch, and public-docstring coverage for the fallback policy.

### Security

- Restrict the private plain-HTTP provider seam to `localhost` or literal loopback IP addresses, reject URL userinfo before connection, dial directly without ambient proxy lookup, reject all redirect responses, and close failed resources deterministically.
Expand All @@ -19,6 +25,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
- Reject malformed UTF-8/JSON, duplicate object names, Python non-finite-number extensions, finite-syntax exponents that overflow Python floats to non-finite values, and non-object top-level values in validated structured provider responses before application parsing; canonicalize valid JSON and strict Batch JSON Lines so later decoder failures cannot retain the original provider document.
- Integrate DNS-pinned provider dispatch directly into `ModelClient` so package import performs no optional-adapter monkey-patching or order-dependent class mutation.
- Reject provider hosts that resolve to any non-globally-routable address, including RFC 6598 shared address space, while retaining explicit multicast, private, loopback, link-local, and reserved-address protections.
- Remove fallback-policy environment-value inspection; trusted callers now declare only validated available credential names, and the policy CLI rejects the former secret-bearing environment selector.
- Reject boolean or floating-point schema versions, unsafe programmatic agent selectors, non-string candidate identifiers, and mutable credential-control collections as controlled validation errors instead of leaking Python type exceptions or permitting post-validation policy changes.
- Document narrowly scoped Semgrep suppressions for parameter-bound database queries, the explicit development-only TLS verification opt-out, and provider URLs that pass the egress guard.

### Changed
Expand Down
23 changes: 23 additions & 0 deletions contextual_orchestrator/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,18 @@
from .cost_router import CostRoutingCoordinator
from .credentials import NotConfigured, get_credential, register_credential
from .kv_config import InMemoryConfigStore, get_config_store
from .model_fallback import (
CandidateValidationError,
CostTier,
FallbackCandidate,
FallbackContext,
FallbackManifestError,
FallbackPlan,
NoEligibleCandidateError,
SkippedCandidate,
build_fallback_plan,
load_fallback_manifest,
)
from .orchestrator import ModelAgent, TaskOrchestrator, WorkflowStep, load_agents
from .token_counting import HeuristicTokenCounter, build_token_counter

Expand Down Expand Up @@ -91,4 +103,15 @@
"build_embeddings_jsonl_body",
"cheapest_upstream",
"CostRoutingCoordinator",
# transport-neutral model fallback policy
"CandidateValidationError",
"CostTier",
"FallbackCandidate",
"FallbackContext",
"FallbackManifestError",
"FallbackPlan",
"NoEligibleCandidateError",
"SkippedCandidate",
"build_fallback_plan",
"load_fallback_manifest",
]
103 changes: 103 additions & 0 deletions contextual_orchestrator/_fallback_cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
"""Command-line adapter for the transport-neutral fallback policy."""

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any, Mapping, Sequence

from ._fallback_manifest import load_fallback_manifest
from ._fallback_plan import build_fallback_plan
from ._fallback_types import (
ALLOWED_VISIBILITIES,
FallbackContext,
FallbackManifestError,
validate_credentials,
)


def _load_manifest_path(path: Path) -> Mapping[str, Any]:
"""Read a UTF-8 JSON manifest and normalize input errors."""
try:
raw = path.read_text(encoding="utf-8")
except OSError as exc:
raise FallbackManifestError(
f"manifest could not be read: {path}"
) from exc
try:
document = json.loads(raw)
except json.JSONDecodeError as exc:
raise FallbackManifestError(
"manifest must contain valid JSON"
) from exc
if not isinstance(document, Mapping):
raise FallbackManifestError("manifest must be an object")
return document


def _declared_credentials(names: Sequence[str]) -> frozenset[str]:
"""Validate and return credential names declared available by the caller."""
normalized = tuple(names)
validate_credentials(normalized)
return frozenset(normalized)


def _build_parser() -> argparse.ArgumentParser:
"""Build the parser for policy-only workflow integration."""
parser = argparse.ArgumentParser(prog="contextual-model-fallback")
subparsers = parser.add_subparsers(dest="command", required=True)
plan_parser = subparsers.add_parser("plan")
plan_parser.add_argument("--manifest", type=Path, required=True)
plan_parser.add_argument("--agent", required=True)
plan_parser.add_argument(
"--repository-visibility",
choices=sorted(ALLOWED_VISIBILITIES),
default="public",
)
plan_parser.add_argument(
"--available-credential", action="append", default=[]
)
plan_parser.add_argument(
"--required-capability", action="append", default=[]
)
plan_parser.add_argument("--deny-paid", action="store_true")
plan_parser.add_argument(
"--format",
choices=("json", "ids", "models"),
default="json",
)
return parser


def main(argv: Sequence[str] | None = None) -> int:
"""Render one validated plan and return an exit code."""
args = _build_parser().parse_args(argv)
if args.command != "plan": # pragma: no cover - argparse constrains it.
raise FallbackManifestError(
f"unsupported command: {args.command}"
)
document = _load_manifest_path(args.manifest)
candidates = load_fallback_manifest(document, args.agent)
context = FallbackContext(
repository_visibility=args.repository_visibility,
available_credentials=_declared_credentials(
args.available_credential
),
required_capabilities=frozenset(args.required_capability),
allow_paid=not args.deny_paid,
)
plan = build_fallback_plan(candidates, context=context)
if args.format == "json":
print(
json.dumps(
plan.to_public_dict(),
sort_keys=True,
separators=(",", ":"),
)
)
elif args.format == "ids":
print(" ".join(plan.candidate_ids))
else:
print(" ".join(candidate.model for candidate in plan.candidates))
return 0
Loading
Loading