diff --git a/AGENTS.md b/AGENTS.md index 7ced99577..e2720ea6e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,6 +38,26 @@ push or open a PR. code-scanning tools can't converge on one PR ref. Gating happens via the Security **job results**; do not add tools to the `code_scanning` rule. +### Repository-writer lease and dependency authority + +- Enforce **one writer per repository branch**. Before every repository write, + refetch the **exact PR head and target blob SHA**. If either changed, inspect + the intervening work and reconcile once before editing; never overwrite an + independently moved branch from stale state. +- Repositories outside `ContextualWisdomLab/contextual-orchestrator`, including + the central `ContextualWisdomLab/.github` control plane and repositories with + their own dedicated maintenance loops, are **read-only dependencies** unless + the task is explicitly assigned to that repository. Do not edit their + branches, dispatch **write-capable agents**, resolve their review threads, or + merge their PRs from this repository's loop. +- Live GitHub state is authoritative. A predecessor-head, stale-head, + cancelled, absent, failed, queued, pending, skipped-required, or + synthetic-merge result is not current-head evidence and must never be reused + to approve or merge a later tree. +- Do not create one-shot, self-modifying, encoded-patch, branch-local repair, or + temporary write-capable GitHub Actions workflows. Prefer direct reviewed + changes tied to the exact current head. + ### Code exploration - This repo has **no `.codegraph/` index**, so use normal search @@ -56,11 +76,10 @@ push or open a PR. - The reference implementation is xtrmLLMBatchPython's pgcrypto-encrypted Postgres credential registry (`get_credential(name)`); reuse that pattern (a DB-backed KV is fine) unless a dedicated KV is adopted. -- **Known deviation to migrate:** this repo currently resolves provider API - keys from env — `ModelClient` reads `os.environ.get(agent.api_key_env)` in - `contextual_orchestrator/orchestrator.py` (and `CONTEXTUAL_ORCHESTRATOR_*` - tokens in `__main__.py`). Move these to KV-backed reads; keep env only as the - bootstrap path that seeds the KV. +- Protected main resolves provider keys through `get_credential`; the legacy + `api_key_env` field is only a credential-name compatibility alias. Do not + reintroduce request-time environment fallback. Process/bind configuration + may still use explicit `CONTEXTUAL_ORCHESTRATOR_*` bootstrap inputs. ### This repo: the org LLM gateway @@ -68,11 +87,15 @@ push or open a PR. OpenAI-compatible front door consumed by **gyeot** and **scopeweave**. - **Direction:** grow it toward a **LiteLLM-class multi-provider gateway**. The org is open to a **Rust/Python hybrid** to cut overhead. -- Its `ModelClient` currently reads `os.environ.get(agent.api_key_env)` — this - is the KV-principle deviation above. Resolve the API key (including the org - `OPENAI_API_KEY`) from the **KV / credential registry**, not env. -- The **OpenCode review pipeline is separate** and stays on **GitHub Models** — - do not change it. +- Its `ModelClient` resolves the credential name through the **KV / credential + registry**, including `OPENAI_API_KEY`; do not add ambient environment + fallback at request time. +- The **OpenCode review pipeline is separate and centrally governed** by + `ContextualWisdomLab/.github`. Do not hard-code or replace its provider pool, + reviewer identities, or credential chain from this repository. For live model + tests and autonomous development work owned by this repository, use + `NVIDIA_NIM_API_KEY`; never repurpose `COPILOT_GITHUB_TOKEN` as a model or + development-agent credential. ### This repo's role in the ecosystem @@ -83,7 +106,7 @@ push or open a PR. email/PIM that DOM-decomposes emails/files into a persisted knowledge graph). Each component below is a **standalone program that must ALSO work as a git submodule**, grown separately and together: - - **waf-ids-ai-soc** — WAF / IDS / AI SOC / LB / APIM. + - **wardnet** — WAF / IDS / AI SOC / LB / APIM. - **clearfolio** — document viewer. - **pg-erd-cloud** — ERD tool. - **contextual-orchestrator** — this repo: LLM cost/perf/upstream-LB gateway @@ -91,7 +114,7 @@ push or open a PR. - **codec-carver** — STT / omni-modal speech-video codec. - **fast-mlsirm** — LLM-as-a-Judge calibration + evaluation-item quality (uses aFIPC FIPC + kaefa item-fit). - - **feelanet-adfs** — passwordless SSO (OIDC/SCIM/ADFS/LDAP/FIDO2/OAuth2.1, + - **keyverse** — passwordless SSO (OIDC/SCIM/ADFS/LDAP/FIDO2/OAuth2.1, eliminate passwords). - **newsdom-api** — PDF→DOM sidecar. - **semantic-data-portal** — upper ontology / catalog / governance plane with @@ -109,3 +132,22 @@ push or open a PR. scheduling (e.g. LLM-cascade / model-routing and queueing/load-balancing papers). + +## Canonical product documentation + +Start at [`docs/README.md`](docs/README.md). Root `ARCHITECTURE.md`, PRD, TRD, +ERD, UML, ADRs, threat model, test strategy, operability, incident response, +traceability, and references are one status-qualified graph. Behavior changes +must update the affected authority and documentation contract test. + +## Execution continuity + +- Treat prompt edits, audits, status summaries, and documentation assessments + as intermediate work when the request also authorizes repository changes. +- Continue the safe chain: verify live target state, repair the smallest + coherent authority set, run focused and full evidence, publish a reviewable + branch/PR, inspect its exact-head state, then take the next non-conflicting + authorized task while a control-plane check is pending. +- Stop only for a real authority choice, destructive ambiguity, permission + boundary, or external dependency that blocks every safe continuation. Never + turn queued, absent, stale, synthetic, or status-only evidence into success. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 000000000..6cd9765b6 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,243 @@ +# Contextual Orchestrator architecture + +**Document state:** `accepted_architecture`
+**Canonical role:** current component, trust-boundary, and deployment authority + +`docs/architecture.md` remains a research-to-product note. This document is the +system architecture authority and links detailed runtime diagrams in +`docs/UML.md` and data ownership in `docs/ERD.md`. + +## Architectural intent + +Contextual Orchestrator is one provider-neutral orchestration domain exposed as +a Python library, CLI, and OpenAI-compatible HTTP service. It keeps policy and +evidence inside one deployable boundary while allowing optional infrastructure +adapters. Basic operation does not require the wider CWL ecosystem. + +```mermaid +flowchart TB + caller["API consumer"] --> delivery["HTTP / CLI delivery"] + operator["Platform operator"] --> admin["Admin and evidence API"] + delivery --> coordinator["CostRoutingCoordinator: sync or batch"] + coordinator --> domain["TaskOrchestrator: route or conduct"] + coordinator --> ledger["Cost ledger"] + coordinator --> batch["Local or pg-llm-batch adapter"] + delivery -. passthrough / route stream .-> domain + admin --> domain + domain --> client["ModelClient provider adapter"] + domain --> state["Optional workflow / agent stores"] + client --> provider["OpenAI-compatible provider"] +``` + +The dotted path is a protected-main exception: raw compatible passthrough and +route streaming bypass part of coordinator accounting. It is a documented gap, +not the target evidence architecture. + +## Bounded contexts + +| Context | Responsibility | Does not own | +|---|---|---| +| Delivery | Authentication, input bounds, HTTP/CLI translation, compatible response framing. | Model policy or provider credentials. | +| Orchestration domain | Route/conduct choice, workflow plan, access lists, agent selection, verification, synthesis, trace, budget. | Host identity, tenant directory, or provider network implementation. | +| Provider adapter | KV credential lookup, compatible request, timeout/retry, usage capture, transport validation. | Workflow policy or review authority. | +| Cost and batch hub | Token/count provenance, configured prices, attribution, sync/batch decision, backend lifecycle. | Route/conduct policy, fabricated prices, or external batch persistence. | +| State and credential adapters | Optional SQLite state/agent overlay, PEP-249 ledger, in-memory or pgcrypto credentials. | Legal basis, tenant authorization, or enterprise backup policy. | +| Operator evidence | Admin, trace, evaluation, access, audit, analytics, and readiness projections. | Certification, independent approval, or production SLO proof. | + +## Module map + +| Module | Role | +|---|---| +| `orchestrator.py` | `ModelAgent`, `WorkflowStep`, `OrchestrationPolicy`, `ModelClient`, `TaskOrchestrator`, state stores, cache, redaction, budgets, traces, and readiness reports. | +| `server.py` | Threaded stdlib HTTP delivery, bearer scopes, validation, rate/concurrency controls, routing, SSE framing, and error translation. | +| `admin.py` | Dependency-free operator console. | +| `api_contract.py` | Machine-readable OpenAPI subset and operation identities. | +| `credentials.py` | Credential protocol, in-memory backend, pgcrypto Postgres backend, and registry functions. | +| `kv_config.py` | Intentional no-DSN in-memory configuration plus an authoritative fail-closed `pg-llm-batch` Postgres adapter on the active #96 stack. | +| `cost_ledger.py` | Price book, prompt-safe usage records, telemetry, non-blocking export, SQL store, and rollups. | +| `batch_routing.py` | Routing hints/policy, local and external chat/embedding batch contracts. | +| `cost_router.py` | Coordinates token counting, sync/batch channel choice, ledger, and backend submission/retrieval. | +| `token_counting.py` | Deterministic heuristic and optional Postgres `pg_tiktoken` adapter. | +| `conventions.py` | Two-or-more-word snake_case validation. | +| `__main__.py` | CLI completion, server, evaluation, and credential bootstrap. | + +## Control plane and data plane + +The control plane includes agent configuration, policy, credentials, prices, +budgets, provider exclusions, evaluation, and operator evidence. The data plane +includes validated request payloads, selected step context, provider requests, +answers, usage signals, and optional batch payload references. + +Control-plane changes may affect later requests but cannot rewrite the evidence +attached to a completed run. Data-plane payloads must not be copied into broad +usage telemetry. Protected main has only admin and inference bearer scopes: no +dedicated trace scope exists, and an inference-scoped caller may request +`include_orchestration_trace: true`. Purpose- and tenant-specific trace authority +is an accepted boundary that still needs host RBAC or a dedicated runtime scope. +Active PR #121 partially removes inference-only disclosure on selected paths, +but it does not yet implement an independent purpose/tenant/resource trace +authority across every trace-bearing surface. + +## Route and conduct + +`TaskOrchestrator.complete()` is the stable split: + +- `route` selects and calls one eligible worker. It is the only mode that can + honestly relay live provider SSE tokens on protected main. +- `conduct` creates a bounded template or validated generated workflow. Each + `WorkflowStep.access` tuple names prior step outputs deliberately included in + that worker's context. Verification precedes synthesis when policy requires + it. Any HTTP stream is framed after the answer exists. + +The deterministic policy is the protected-main authority. Learned routing, +recursive coordination, and role-specific reasoning controls require +comparable-budget evidence before replacing it. + +Protected-main agent choice is deterministic tag/domain/priority scoring. It is +not learned, price-aware, or load-balanced. `route_p95_seconds` is exposed but +does not currently participate in dispatch, and `cheapest_upstream()` is not +called by either routing layer. + +## Trust boundaries + +1. **Caller boundary:** bearer scope, bind policy, body/role/mode/rate/concurrency + validation precede orchestration. +2. **Context boundary:** access lists limit cross-step visibility. Trace exposure + defaults off, but protected-main inference authority can opt in; dedicated + purpose/tenant trace RBAC remains `planned`. +3. **Credential boundary:** provider secrets are names in model configuration + and values in KV; environment is bootstrap transport only. +4. **Provider boundary:** protected main requires HTTPS and globally routable + destinations. The stronger DNS-pinned, redirect/proxy-safe, strictly bounded + response implementation is `active_pr` in #96. +5. **Persistence boundary:** in-memory is default. Enabling a file or database + creates an operator obligation for access, encryption, retention, backup, + deletion, and recovery. +6. **Evidence boundary:** a local report, check status, automated review, human + approval, and protected merge are different authorities. +7. **Host boundary:** a CWL host retains identity, tenancy, legal basis, + business data, and deployment unless a versioned contract delegates them. + +## Data ownership + +- In-memory workflow, evaluation, audit, analytics, circuit, and cache state are + process-owned and ephemeral. +- Optional SQLite stores provide standalone durability, not a normalized + enterprise data plane. +- The cost ledger has an in-memory default and a portable PEP-249 SQL store. +- The active `PriceBook` reads ConfigStore, not the SQL + `llm_price_entries` table. That table is created but dormant. +- Provider credentials may be in-memory for development or pgcrypto-encrypted + in Postgres. +- `docs/database_design.sql` is a normalized production target and must not be + confused with runtime-created SQLite schemas. +- External batch/config/secret objects accessed through `pg-llm-batch` are + owned by that service or adapter. + +## Deployment forms + +### Standalone + +One process serves CLI or HTTP, mock or configured providers, in-memory state, +and optional SQLite/SQL/KV adapters. Loopback binding is the safe default. + +### CWL composition + +An ingress or host authenticates the user and supplies a purpose-bound request. +Contextual Orchestrator selects and executes models. `pg-llm-batch` may execute +latency-tolerant work. naruon, inkspan, Clearfolio, and other systems consume +explicit interfaces and retain their own data and authorization boundaries. + +## Failure domains and degraded behavior + +| Domain | Isolation and degraded behavior | +|---|---| +| One provider/model | Bounded transient client retry and eligible orchestration failover; permanent provider errors receive no same-client retry and fail when no eligible candidate remains. | +| Credential registry | Non-mock execution fails closed; mock/offline operation remains available. | +| Optional state store | Persistence evidence is unavailable; the service must not claim durable history. | +| Cost export | Non-blocking store may degrade while prompt-safe health exposes the loss. | +| External batch service | Interactive route remains independently usable; process-local job lookup is lost on restart even when an external job survives. | +| Admin integration | Inference and library paths remain independently usable. | +| Automated review/control plane | Protected merge waits; repository-local development and verification continue. | + +## Architecture invariants + +- Agent pools are data, not provider-specific branches in domain logic. +- Access is explicit; a worker never receives all previous outputs by default. +- Credentials are resolved by name at the provider boundary. +- Estimates are labeled and unknown prices remain unknown. +- Optional integrations do not break standalone behavior. +- No repository-local result claims certification or independent approval. +- New scientific arithmetic owned by this service is Rust-first with + parity-verified CPU/GPU paths; currently such arithmetic is `out_of_scope`. +- Database identifiers use two-or-more-word snake_case unless an external + standard fixes the field name. + +## Known protected-main divergences + +- Workflow-derived spend/budget and the independent cost ledger are not + synchronized. Missing ledger price becomes zero while spend analytics labels + it unknown; this violates the accepted unknown-price invariant. +- Raw passthrough records analytics but no workflow or ledger row. Route + streaming bypasses the coordinator and durable `_StateStore`; a mid-stream + failure can leave no retained run. +- Coordinator batch handles are process-local. Restart loses lookup, and chat + result replay can duplicate usage; embedding idempotency is also process-local. +- Static OpenAPI, runtime dispatch, scopes, and endpoint prose are separate + authorities and have drifted. +- Protected `main` may still downgrade a configured Postgres configuration path + to process-local memory. The `active_pr` #96 stack used by this documentation + branch changes that behavior: An explicitly configured Postgres KV backend is + authoritative and fails closed with ConfigBackendUnavailableError. This is + not protected-main behavior until #96 merges. +- Token counting may deliberately degrade to the documented heuristic when the + optional Postgres counter cannot be constructed; that result remains + estimated evidence and must be operator-visible. +- Protected main combines liveness/readiness detail, accepts incomplete inbound + framing states, and couples trace disclosure to broad bearer scopes. Active + PR #121 is a partial hardening slice, but duplicate Content-Length, + transfer-coding rejection, body deadlines/desynchronization, independent + trace authority, and real dependency-readiness degradation remain incomplete. +- Commercial/readiness responses are derived documents, not persisted domain + entities or external attestations. + +## Status-qualified evolution + +- PR #96: `active_pr` provider transport, response trust, configured-KV + fail-closed boundary, and portable Atheris prerequisite; source-complete but + not protected or independently approved. +- PR #109: `active_pr` local loopback MLX provider and audited model judgment; + coverage, ancestry, structured review, and independent approval remain + unprotected evidence. +- PR #111: `active_pr` partial price-aware tie-breaking, administrator + credential, and opaque-session slice; Secure-cookie, CSRF/origin, + bounded-session, restart/durability, disclosure-sink, security-base, and + approval blockers remain. +- PR #112: `active_pr` fail-closed evidence-model prototype; caller-supplied + dictionaries are not a trusted protected-head release-authority binder. +- PR #114: `active_pr` partial immediate-race experiment; explicit equivalence, + completed-response validation, cancellation/drain, budget, accounting, + deterministic tie-breaking, delayed hedge, and ablation acceptance remain + incomplete. +- PR #115: open `superseded` NIM catalog scaffold; useful bounded discovery + evidence does not satisfy issue #86's security, modality, benchmark, cost, + uncertainty, provenance, and transactional-artifact contract. +- PR #121: `active_pr` open partial liveness/readiness, request-framing, and + trace-authority slice. Issues #117, #118, and #119 remain open and incomplete. +- PR #66: `superseded` closed-unmerged synchronous-embeddings and KV-bootstrap + evidence; the requirement remains planned. +- PR #82: `superseded` closed-unmerged dependency-bootstrap evidence; rebuild + only its unique intent after #96 protects main. +- PR #90: `superseded` closed-unmerged NIM benchmark evidence; issue #86 + remains planned. +- PR #94: `superseded` closed-unmerged free-first fallback evidence; the + requirement remains planned. +- PR #99: `superseded` closed-unmerged adaptive-reasoning evidence; the + requirement remains planned. +- PR #113 and PR #120: `superseded` closed-unmerged duplicate documentation + replays; accepted unique disclosure and canonical-graph intent is retained in + PR #105 rather than a second authority. + +No active pull request is architecture authority until its exact head passes +repository policy and reaches protected main. See `docs/TRACEABILITY.md` for the +status-qualified relationship graph and `docs/adr/README.md` for decision status. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a04eea25..f2a90e610 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,24 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and ### Documentation +- Reconcile the live implementation inventory by classifying closed-unmerged PR #66, PR #82, PR #90, PR #94, and PR #99 as `superseded`, recording current active slices without promoting them to protected authority, and separating protected-main configuration fallback from PR #96's configured-Postgres fail-closed behavior and deliberate heuristic token-count evidence. +- Add an indexed continuation evidence appendix that preserves the original + documentation audit, classifies exact-head, integration, review, and absent + evidence, and records protected-main release gaps without promoting active + pull requests to shipped authority. +- Separate durable requirement traceability from volatile SHA, workflow, + review, and branch snapshots by moving the dated audit into an indexed + evidence appendix and enforcing that boundary in documentation fitness tests. +- Align the canonical cost, access-grant, internationalization, research-license, coverage, failure-flow, credential-authority, and status contracts with exact-head automated review findings and machine-check them. +- Status-qualify the analytics, REST API, and internationalization guides, replacing legacy prototype labels and distinguishing current standalone paths from optional planned framework adoption. +- Align Claude and conductor guidance with the current provider-neutral product and status-qualified dependency-adoption boundary, removing legacy lab and internal gate names. +- Replace stale lab/prototype and internal-name language in library research with machine-checked current-stack and adoption-status boundaries for the stdlib HTTP/admin path and optional API/database extras. +- Correct the supporting product plan's enterprise-auth boundary: the standalone runtime has coarse admin/inference bearer scopes, not tenant-aware RBAC, while the host owns enterprise identity and tenancy. +- Remove the remaining stdlib-lab qualifier from spend observability and describe the evidence boundary as a standalone deployment without promoting local signals to billing or compliance evidence. +- Replace the competitor-centric README disclaimer with an affirmative independent-implementation, third-party-model-weight, proprietary-artifact, and provider-boundary statement for commercial provenance review. +- Replace legacy lab framing in the root README with the current buyer-facing provider-neutral orchestration-control-plane identity and local-deployment boundary. +- Add a canonical release, migration, and rollback guide that binds protected-source identity, reproducible build and artifact provenance, state migration, publication, rollback, and protected-main operational acceptance without presenting Draft evidence as shipped. +- Establish a canonical status-qualified product documentation graph spanning PRD, TRD, architecture, UML, ERD, ADRs, threat model, test strategy, operability, incident response, traceability, standards/research references, and machine-checked authority boundaries without promoting active or planned work as shipped. - Add APA 7 doctoring for Python environment-marker semantics, Atheris artifact availability and hashes, and the supported-platform uncertainty boundary. - Add provider-response resource-bound doctoring covering the 8 MiB fail-closed limit, HTTP framing preflight, `text/event-stream` media-type enforcement, bounded SSE reads, OpenAI-compatible `[DONE]` completion evidence, malformed-event and premature-EOF handling, batch-output partitioning, incident handling, and operational rollback. - Add provider-stream UTF-8 doctoring grounding strict SSE/JSON decoding and redacted malformed-input handling in the WHATWG HTML Standard and RFC 8259, with verification, failure, rollback, and authority boundaries. diff --git a/CLAUDE.md b/CLAUDE.md index f893b5f7b..abf4d92f9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,12 +7,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co `AGENTS.md` is the canonical, tool-agnostic agent operating guide for this repo. Read it fully and follow its guardrails before making changes. In particular: - **Security gate**: every PR to `main` runs the required Security workflow. A failing Trivy or pip-audit job is a real finding — remediate by bumping the dependency and regenerating `requirements.lock`; never weaken, `continue-on-error`, or disable the gate. +- **Writer lease**: enforce **one writer per repository branch**. Before every write, refetch the **exact PR head and target blob SHA**. Treat central `.github` and repositories with their own maintenance loops as **read-only dependencies**; do not edit them, dispatch **write-capable agents**, resolve their threads, or merge their PRs from this repository. Never reuse predecessor-head, stale-head, queued, pending, absent, failed, or synthetic-merge evidence. - **KV, not env**: runtime config and provider secrets are resolved from the KV credential registry (`get_credential`), never `os.getenv` at request time. Env is only bootstrap transport into the KV (see `docs/kv-credentials.md`). -- **Org role**: this repo is the org's LLM gateway (cost optimizer + sync/batch routing + upstream load balancing, LiteLLM-plus scope), consumed by `gyeot` and `scopeweave`. The OpenCode review pipeline is separate, stays on GitHub Models, and must not be changed. +- **Org role**: this repo is the org's LLM gateway (cost optimizer + sync/batch routing + upstream load balancing, LiteLLM-plus scope), consumed by `gyeot` and `scopeweave`. The OpenCode review pipeline is separate and centrally governed by `ContextualWisdomLab/.github`; do not replace its provider pool, reviewer identities, or credential chain from this repository. Repository-owned live model tests and autonomous development work use `NVIDIA_NIM_API_KEY`, never `COPILOT_GITHUB_TOKEN`. - **Research grounding**: substantive feature/process PRs should attach the relevant papers (PDF when redistribution is permissible, otherwise cite + link + summary) under `docs/papers/` with full citations. This file complements AGENTS.md with commands and architecture; where they differ, AGENTS.md wins. +Canonical product and engineering authority is indexed at +[`docs/README.md`](docs/README.md). Treat PR bodies, conversation history, and +commercial evidence packets as supporting evidence rather than substitutes for +the status-qualified PRD, TRD, architecture, ERD/UML, ADRs, security, test, and +operability documents. + ## Common commands ```bash @@ -70,7 +77,12 @@ CI gates: `.github/workflows/security.yml` (CodeQL + pip-audit on `requirements. ## What this is -A stdlib-Python lab implementing a single OpenAI-compatible API that routes, delegates, verifies, and synthesizes work across a configurable pool of model agents — plus the org's cost-review and sync-vs-batch routing hub. Runtime dependencies are the Python standard library only (Hypothesis is the sole listed dependency, for the property tests); FastAPI/SQLAlchemy/psycopg exist as *optional* extras for the hardened production target, not the current runtime. +A provider-neutral OpenAI-compatible orchestration control plane that routes, +conducts, verifies, and synthesizes work across governed model agents, plus the +organization's cost-review and sync-versus-batch routing hub. The current HTTP +and control path uses the Python standard library. The optional `api` and `db` +extras are installable compatibility surfaces; they do not establish FastAPI, +SQLAlchemy ORM, or Alembic as owners of a current production call path. ## Architecture @@ -87,7 +99,7 @@ A stdlib-Python lab implementing a single OpenAI-compatible API that routes, del - `orchestrator.py` — the domain heart: `ModelAgent`, `WorkflowStep`, `OrchestrationPolicy`, `ModelClient`, `TaskOrchestrator`, secret/PII redaction, budget enforcement, spend analytics, and the commercial-readiness report generators behind `/api/v1/*`. Domain code stays here until a second implementation forces extraction (see `docs/code_conventions.md`). - `server.py` — HTTP delivery adapter and `SecurityConfig`; all request validation lives here. -- `admin.py` — static HTML/CSS/JS for the `/admin` operator console (stays inline while the product is dependency-free). +- `admin.py` — static HTML/CSS/JS for the current `/admin` operator console. - `credentials.py` / `kv_config.py` — the KV seam: `get_credential`/`register_credential` over pluggable backends (`InMemoryCredentialBackend` default; pgcrypto-encrypted `PostgresCredentialBackend`, selected via `CONTEXTUAL_ORCHESTRATOR_KV_BACKEND`). - `cost_ledger.py` / `cost_router.py` / `batch_routing.py` / `token_counting.py` — the cost-review + routing hub: prompt-safe usage ledger with seven attribution dimensions, `RoutingPolicy` (sync vs batch from request hints + KV thresholds), and the [pg-llm-batch](https://github.com/ContextualWisdomLab/pg-llm-batch) batch/embeddings backends (a local in-process backend keeps the standalone path working with no external service). - `api_contract.py` / `conventions.py` — API-shape and naming-rule enforcement helpers. @@ -97,12 +109,12 @@ Agent pools are **data, not code**: `examples/agents.mock.json` and `examples/ag ### `conductor/` — context, not code -`conductor/` is the CDD (context-driven development) directory, not a Python package: `product.md` (intent and non-goals), `tech-stack.md` (stdlib-only rationale), `workflow.md` (the TDD/DDD/CDD method and the Ponytail design gate), `tracks.md` (active tracks). Update it when scope, dependencies, workflow, or domain terms change. +`conductor/` is the CDD (context-driven development) directory, not a Python package: `product.md` (intent and non-goals), `tech-stack.md` (current and planned dependency status), `workflow.md` (the TDD/DDD/CDD method and dependency-adoption gate), `tracks.md` (active tracks). Update it when scope, dependencies, workflow, or domain terms change. ## Key conventions - **TDD from papers**: paper claims (Fugu, TRINITY, Conductor — see `docs/architecture.md`) become executable contracts in `tests/` *before* implementation changes. Many tests assert doc/API contracts, so behavior changes usually require updating the matching `docs/*.md` in the same PR. - **Naming**: configurable, API, and DB object names must be lower snake_case with **two or more semantic words** (`agent_pool`, `workflow_run`; never `agent` or `agentPool`). Enforced by `conventions.require_object_name()` and `tests/test_conventions.py`. Paper role values (`thinker`, `worker`, `verifier`, `synthesizer`) are deliberate exceptions. -- **Ponytail design gate**: before adding a dependency or designing a subsystem, research existing libraries and record the decision in `docs/library_research.md`. No new dependency when the stdlib or an already selected library covers the need; no interface or factory until a second real implementation exists. +- **Dependency-adoption gate**: before adding a dependency or designing a subsystem, research existing libraries and record the decision in `docs/library_research.md`. No new dependency when the standard library or an already selected library covers the need; no interface or factory until a second real implementation exists. - **Honest metrics**: spend/analytics surfaces label estimates (`usage_source`, `measurement_status`) and never fabricate prices — preserve this when touching analytics. - **Fuzz seams**: untrusted-input parsers (request body, agent config, redaction, orchestration) share invariant checks in `fuzz/targets.py`, driven by both Hypothesis (`tests/fuzz/`) and Atheris (`fuzz/`). New parsing seams should get a target there (see `docs/fuzzing.md`). diff --git a/README.md b/README.md index 65f57dd4c..56e8890e5 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,18 @@ [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/contextual-orchestrator) [![Security](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/workflows/security.yml/badge.svg)](https://github.com/ContextualWisdomLab/contextual-orchestrator/actions/workflows/security.yml) -Stdlib Python lab for a single API that routes, delegates, verifies, and synthesizes work across a configurable pool of OpenAI-compatible model agents. +Provider-neutral OpenAI-compatible orchestration control plane that routes, +conducts, verifies, and synthesizes work across governed model agents. -This is not a Sakana AI product or a reproduction of their trained models. It is a small implementation of the public architecture pattern: expose one model-like interface while keeping the agent pool, routing, workflow, and verification logic behind it. +Contextual Orchestrator is independently implemented from published +orchestration concepts. It includes no third-party trained model weights or +proprietary artifacts; operators supply models through provider-neutral +OpenAI-compatible endpoints. + +Product, technical, security, data-model, operational, and decision authority +is indexed in [docs/README.md](docs/README.md). Capability status is qualified +there as shipped on protected main, active-PR, accepted architecture, planned, +research-only, superseded, or externally owned. ## Quick Start @@ -34,15 +43,15 @@ curl -s http://127.0.0.1:8000/v1/chat/completions \ -d '{"model":"contextual-orchestrator","messages":[{"role":"user","content":"Analyze this code review task and verify the answer."}]}' | jq . ``` -HTTP serving is hardened for local lab use: +HTTP serving is hardened for local deployment: - `/admin`, `/admin/state`, `/api/v1/*`, and `/v1/chat/completions` require a Bearer token. Use `--admin-token` and `--inference-token` to separate operator and runtime access, or `--auth-token` / `CONTEXTUAL_ORCHESTRATOR_TOKEN` for one local-development token. - Binding to `0.0.0.0` or `::` requires `--allow-public-bind`. - JSON request bodies, chat message roles, orchestration modes, body sizes, request rate, and concurrent run counts are validated before orchestration runs. -- Full orchestration traces are not returned by default. Set `include_orchestration_trace: true` per chat request or start with `--expose-trace-by-default` when the caller is trusted. -- State is in-memory by default. Pass `--state-db PATH` (or `CONTEXTUAL_ORCHESTRATOR_STATE_DB`) to persist workflow runs, evaluation runs, audit, and analytics to a stdlib sqlite file so they survive a restart; without it, behavior is unchanged. +- Full orchestration traces are not returned by default. Set `include_orchestration_trace: true` per chat request or start with `--expose-trace-by-default` when the caller is trusted. Protected main has no dedicated trace scope: any inference-scoped caller can opt in, so isolate that token at the host/gateway boundary. +- State is in-memory by default. Pass `--state-db PATH` (or `CONTEXTUAL_ORCHESTRATOR_STATE_DB`) to persist ordinary non-stream workflow runs, evaluation runs, audit, and analytics to a stdlib sqlite file so they survive a restart. Route-stream workflow runs remain memory-only, and compatible passthrough creates no workflow run. - Response caching is off by default. Pass `--cache-ttl SECONDS` to serve identical requests (same messages + mode) from an in-memory TTL+LRU cache and skip the provider calls; `0` disables it. -- `ModelClient.batch_chat(agent, {custom_id: messages})` runs many requests through the provider's Batch API (async, 24h completion window, typically ~50% cheaper) — suited to evaluation/benchmark workloads, not latency-sensitive chat. The mock path answers synchronously. +- `ModelClient.batch_chat(agent, {custom_id: messages})` runs many requests through a provider-native Batch API. Completion windows and discounts are provider-contract facts, not repository guarantees. The mock path answers synchronously. Use real workers by replacing `mock://` agents with OpenAI-compatible endpoints. Provider secrets are resolved from a KV credential registry via `get_credential`, never from `os.getenv` at request time (see [docs/kv-credentials.md](docs/kv-credentials.md)): @@ -62,13 +71,20 @@ Use real workers by replacing `mock://` agents with OpenAI-compatible endpoints. The agent pool is manageable at runtime: `POST`/`PATCH`/`DELETE` on `/api/v1/agent_pools/default/worker_agents[/{id}]` add, govern, and remove model-group members. Pass `--agents-db PATH` (or `CONTEXTUAL_ORCHESTRATOR_AGENTS_DB`) to persist those changes to a stdlib sqlite file — stored changes overlay the seed agents file at startup, and removals write disabled tombstones so they survive restarts; without it the pool is in-memory as before. -Seed the credential into the KV once at bootstrap: +For a one-shot bootstrap that survives process exit, configure the Postgres KV +backend first, then seed the credential: ```bash echo "$OPENAI_API_KEY" | python -m contextual_orchestrator register-credential --name OPENAI_API_KEY --value-stdin ``` -Non-mock providers must use `https://` URLs and a **resolvable KV credential** — a non-mock agent whose credential is missing raises `NotConfigured` rather than falling back to an environment variable. The runtime blocks loopback, private, link-local, multicast, and reserved provider addresses before sending a key. Set `CONTEXTUAL_ORCHESTRATOR_ALLOWED_PROVIDER_HOSTS` to a comma-separated host allowlist when only approved model gateways should be reachable. External calls use a timeout and default output token cap. +The default in-memory credential backend is process-local: running that command +without the Postgres bootstrap variables writes only to the short-lived CLI +process and cannot seed a separately started server. For memory-backed dev/tests, +call `register_credential(...)` inside the same long-lived Python process. See +[docs/kv-credentials.md](docs/kv-credentials.md) for the persistent bootstrap. + +Non-mock providers must use `https://` URLs and a **resolvable KV credential** — a non-mock agent whose credential is missing raises `NotConfigured` rather than falling back to an environment variable. The runtime blocks loopback, private, link-local, multicast, and reserved provider addresses before sending a key. Set `CONTEXTUAL_ORCHESTRATOR_ALLOWED_PROVIDER_HOSTS` to a comma-separated host allowlist when only approved model gateways should be reachable. External calls use a socket timeout and send a default `max_tokens` request hint; protected main does not enforce a provider-response byte or cumulative-SSE cap. > The legacy `api_key_env` field is still accepted for back-compat, but its value is now treated as the **credential name** in the KV, not as an environment variable to read. This supersedes the old `api_key_env` env pattern. @@ -115,9 +131,11 @@ One fused orchestration loop: - Deep path: a natural-language workflow is built with planner, worker, verifier, and synthesizer steps. - Each step has an access list, so workers see only the prior outputs intentionally exposed to them. - Agent definitions are data, so provider preference, exclusions, privacy constraints, and mock testing do not require code changes. -- Provider calls are resilient: transient failures (timeouts, 429, 5xx) retry with full-jitter exponential backoff, while caller errors (4xx) fail fast. If an agent still fails, the request fails over to the next capability-matched agent in the pool, and a per-agent circuit breaker skips a persistently failing provider until it cools down. Failover is recorded in the trace (`served_agent_id`, `failover_from`). +- On ordinary non-stream orchestrated calls, transient failures (timeouts, 429, 5xx) retry with full-jitter exponential backoff, while caller errors (4xx) fail fast. If an agent still fails before output is emitted, the request fails over to the next capability-matched agent, and a per-agent circuit breaker skips a persistently failing provider until it cools down. Failover is recorded in the trace (`served_agent_id`, `failover_from`). Route streaming cannot cross-agent fail over after emission; compatible passthrough and batch paths follow their own provider/backend behavior. -See [docs/architecture.md](docs/architecture.md) for the source-backed analysis. +See [ARCHITECTURE.md](ARCHITECTURE.md) for the current system authority and +[docs/architecture.md](docs/architecture.md) for the source-backed research +mapping. ## Observability & spend @@ -137,19 +155,21 @@ curl -s http://127.0.0.1:8000/api/v1/spend_analytics/latest \ --budget-max-output-tokens 2000000 --budget-max-cost-usd 50 ``` - Or in code: `TaskOrchestrator(budget_max_output_tokens=..., budget_max_cost_usd=...)`. Once spend reaches a cap, the next run is refused — `run()` raises `BudgetExceededError` and `/v1/chat/completions` returns HTTP `429 budget_exceeded`. Current state is in `spend_analytics()["budget"]` (`enabled`, limits, `spent_*`, `remaining_*`, `exceeded`). Cost caps require a price table; token caps do not. + Or in code: `TaskOrchestrator(budget_max_output_tokens=..., budget_max_cost_usd=...)`. For requests that execute through `TaskOrchestrator.run()`, once accounted spend reaches a cap, the next precheck is refused — `run()` raises `BudgetExceededError` and the ordinary coordinator-backed `/v1/chat/completions` path returns HTTP `429 budget_exceeded`. Current state is in `spend_analytics()["budget"]` (`enabled`, limits, `spent_*`, `remaining_*`, `exceeded`). The budget precheck is process-local and non-atomic; compatible passthrough and route streaming bypass workflow-spend advancement, so this is not a universal provider quota. Cost caps require a price table; token caps do not. - **Admin.** The `/admin` **Observability** view renders the totals and the per-model table (unpriced models show an `unpriced` chip). -These are process-local measured signals for a stdlib lab, not a billing system or production compliance data. +These are process-local measured signals for a standalone deployment, not a billing system or production compliance data. ## Cost review + routing hub -The orchestrator is the single control point for **LLM cost review** and -**sync-vs-batch routing** (a LiteLLM-plus scope: cost optimiser + upstream load -balancing + batch routing). All config — prices, thresholds, batch endpoints — -is read from a **KV config store**, never `os.getenv`. +`CostRoutingCoordinator` owns **sync-vs-batch routing** and its independent +ledger; `TaskOrchestrator` separately owns **route-vs-conduct**, deterministic +agent selection, and workflow-derived spend/budget. Routing prices, thresholds, +and batch endpoints come from ConfigStore. Process/bind/bootstrap settings still +have explicit environment/CLI inputs. -- **Usage + cost ledger.** Every completion, sync *and* batch, builds a +- **Usage + cost ledger.** Ordinary coordinator sync completions and retrieved + batch results build a prompt-safe usage record with generated IDs, token counts, cost, provider, model, channel, route mode, and attribution dimensions. Raw prompt and answer text are not part of the usage record or telemetry event. The default @@ -161,6 +181,10 @@ is read from a **KV config store**, never `os.getenv`. service, upstream API/provider, model name, team, group, company**. Token counts reuse `pg-llm-batch`'s `pg_tiktoken` counter when a Postgres DSN is configured, and fall back to a deterministic heuristic otherwise. + Raw passthrough and route streaming currently bypass this ledger. Missing + ledger prices become zero, while workflow spend reports them as unknown; the + two authorities are not yet reconciled. This is not a billing or free-routing + claim. - **Reporting.** `GET /api/v1/cost_reports/rollup?dimension=team&start=&end=` rolls up cost + tokens by any dimension over any time window; `GET /api/v1/llm_usage_records` lists raw ledger rows; @@ -177,6 +201,8 @@ is read from a **KV config store**, never `os.getenv`. Submit via `POST /api/v1/batch_routing_jobs`, poll `GET /api/v1/batch_routing_jobs/{id}`, retrieve `POST /api/v1/batch_routing_jobs/{id}/results` (which records usage + cost). + Coordinator handles are process-local and chat result replay is not yet + idempotent across restart. - **Batch embeddings.** Bulk, latency-tolerant embedding work (e.g. naruon's email-import backfill) submits to `POST /v1/batch/embeddings` (`{model, input|inputs:[...], endpoint, metadata|attribution}`) and polls diff --git a/SECURITY.md b/SECURITY.md index feaf41234..392673456 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,18 +1,66 @@ # Security Policy +This policy defines the public vulnerability-reporting and coordinated-disclosure boundary for `ContextualWisdomLab/contextual-orchestrator`. It is informed by ISO/IEC 29147:2018 for vulnerability disclosure and ISO/IEC 30111:2019 for vulnerability handling. The evidence basis and review dates are recorded in `docs/doctoring/security-disclosure-lifecycle.md`. + +## Supported Versions + +No stable release currently exists, and `main` is not a supported release. +When the first stable release is published, this section will name its supported +version or release line. + +Security fixes are prepared for the latest supported release and, when a vulnerability materially affects an older release that is still explicitly supported, for that supported line as well. Development branches, historical tags, archived artifacts, forks, and unreleased commits are not represented as supported production versions merely because they remain accessible. + +When no stable release has been published, `main` is the integration reference but is not itself a release-support promise. A GitHub Security Advisory or release advisory is the authoritative place to identify affected and patched versions for a specific vulnerability. + +## Scope + +In scope are vulnerabilities in this repository's maintained source, packaging, release artifacts, provider-neutral orchestration interfaces, authentication and credential boundaries, network egress controls, persistence boundaries, and first-party GitHub Actions workflows. + +Reports about third-party services or dependencies are useful when they demonstrate an impact on this project, but upstream-only defects should normally be reported to the responsible upstream maintainer. Findings in unrelated ContextualWisdomLab repositories should be reported through those repositories' own security channels. Do not use a vulnerability report as authorization to test third-party infrastructure, access data that is not yours, degrade service, or bypass provider terms. + ## Reporting a Vulnerability -Report suspected vulnerabilities through GitHub private vulnerability reporting for `ContextualWisdomLab/contextual-orchestrator`: +Use GitHub private vulnerability reporting for `ContextualWisdomLab/contextual-orchestrator` whenever it is available: https://github.com/ContextualWisdomLab/contextual-orchestrator/security/advisories/new +A useful report includes the affected component and version or commit, prerequisites, reproducible steps, observed impact, expected impact boundary, and any safe proof-of-concept material needed to validate the issue. Remove credentials, personal data, private model reasoning, and unrelated customer or provider data. + If private reporting is unavailable, open a public issue that contains only a request for a secure disclosure channel. Do not include exploit details, secrets, personal data, or unreleased vulnerability details in a public issue. -## Response Process +Before any stable release, maintainers must verify that private vulnerability +reporting is enabled and that security-notification recipients are configured. +If that private channel cannot be maintained, release authorization remains +blocked until this policy names a monitored alternative private contact. + +## Coordinated Disclosure Lifecycle + +1. **Receive and acknowledge.** Maintainers triage a private report and aim to acknowledge a credible report within 5 business days. This acknowledgement target is a communication objective, not a remediation SLA and not a promise that validation or a fix will complete within five days. +2. **Validate and scope.** Maintainers reproduce the report where practical, classify affected versions and deployment assumptions, identify downstream or multi-vendor coordination needs, and keep unpatched technical details private. +3. **Remediate and verify.** A fix is developed through a private security collaboration or another access-controlled path when premature disclosure would increase risk. Security-sensitive fixes must retain the repository's tests, coverage, security scanning, provenance, branch-protection, and independent-review requirements rather than bypass them. +4. **Coordinate release.** Maintainers and the reporter coordinate a disclosure point that reasonably allows a verified fix or mitigation to be available. Multi-vendor issues may require additional coordination time. +5. **Publish evidence.** When disclosure is appropriate, publish a GitHub Security Advisory and release or upgrade guidance that identifies affected versions, impact, remediation or mitigation, and patched versions. Request a CVE through the applicable advisory process when warranted and available. +6. **Learn and prevent recurrence.** Record the relevant root-cause class, regression evidence, and preventive control without publishing credentials, private data, or unnecessary exploit-enabling detail. + +Reporter credit is offered when requested and appropriate, subject to the reporter's preference, coordinated-disclosure needs, and GitHub advisory capabilities. A reporter may also request not to be credited. + +## Safe Harbor and Research Boundaries + +We support good-faith security research that stays within the scope above, avoids privacy violations and service degradation, uses the minimum access needed to demonstrate the issue, stops when unintended sensitive data is encountered, and gives maintainers a reasonable opportunity to remediate before public disclosure. This policy does not authorize activity against third-party systems, physical systems, accounts or data you do not control, or conduct prohibited by applicable law or provider terms. + +Do not intentionally persist, download, modify, or disclose data that is not yours. Do not perform denial-of-service testing, social engineering, credential stuffing, destructive testing, or high-volume automated probing against production services. If testing unexpectedly exposes sensitive information, stop, preserve only the minimum evidence needed to report the issue, and disclose it privately. + +## Advisory and Release Evidence + +A vulnerability is not considered remediated merely because a patch exists on a branch. The [canonical release guide](docs/RELEASE_GUIDE.md) governs the release-evidence states. Release evidence must identify the exact integrated revision and the released revision and must not treat queued, pending, skipped-required, cancelled, failed, absent, stale-head, predecessor-head, author-only, status-only, synthetic-merge-only, rate-limited, or infrastructure-only evidence as passing. Security advisories should identify the affected and patched version ranges and link to release or upgrade guidance when practical. + +This policy does not replace repository merge policy: qualifying independent review, unresolved-finding disposition, required checks, branch protection, packaging, provenance, and release-acceptance controls remain authoritative for security releases. -- A maintainer should acknowledge a valid private report within 5 business days. -- Security fixes should be handled on a private branch until the patch is ready to publish. -- Public disclosure should include affected versions, impact, mitigation, and upgrade guidance. +The canonical [threat model](docs/THREAT_MODEL.md) defines assets, trust zones, +abuse cases, controls, and residual risk. The [incident +runbook](docs/INCIDENT_RUNBOOK.md) defines triage, containment, evidence +preservation, recovery, and post-incident acceptance. Neither document claims +an external certification or attestation. ## Automated Checks diff --git a/conductor/tech-stack.md b/conductor/tech-stack.md index 18bbff6d6..fc20230b9 100644 --- a/conductor/tech-stack.md +++ b/conductor/tech-stack.md @@ -4,16 +4,21 @@ Python 3.11+. -## Dependencies +## Current implementation dependencies -Runtime dependencies: none beyond the Python standard library. +The current Python HTTP and control path uses the standard library. Project +metadata also declares optional `api` and `db` extras. Those optional extras are +installable compatibility surfaces, not proof of an implemented framework or +ORM integration. -Production target dependencies after this lab hardens: +## Planned adoption candidates -- FastAPI for REST API, OpenAPI, typed request/response validation, and dependency injection. -- React-admin for the enterprise admin console. -- i18next for shared web i18n. -- PostgreSQL, SQLAlchemy, and Alembic for persistence and migrations. +- FastAPI for a future typed REST adapter when its migration triggers are met. +- React-admin for a separately built enterprise admin client. +- i18next for shared web internationalization. +- SQLAlchemy and Alembic for a future normalized persistence layer. Current + code uses SQLite/PEP-249 state paths and an optional direct-psycopg credential + backend. ## Interfaces @@ -33,4 +38,6 @@ DDD, kept minimal: ## Rationale -The current goal is to encode architecture and workflow contracts, not provider-specific ergonomics. Add FastAPI, OpenAI SDKs, async workers, or persistent storage only after tests show the stdlib version is the bottleneck. +Adopt FastAPI, provider SDKs, async workers, or an ORM only after a bounded +product requirement, migration and rollback evidence, and tests prove the new +authority boundary. diff --git a/conductor/workflow.md b/conductor/workflow.md index 40712b219..83f2db686 100644 --- a/conductor/workflow.md +++ b/conductor/workflow.md @@ -37,7 +37,7 @@ No interface or factory until a second real implementation exists. Context lives under `conductor/`. Update it when scope, dependencies, workflow, or domain terms change. -## Ponytail Design Gate +## Dependency-adoption gate Before adding or designing a subsystem, research existing libraries first and record the decision in `docs/library_research.md`. diff --git a/docs/ERD.md b/docs/ERD.md new file mode 100644 index 000000000..85ede0423 --- /dev/null +++ b/docs/ERD.md @@ -0,0 +1,191 @@ +# Data model and ERD + +**Document state:** `accepted_architecture` + +This document separates actual protected-main storage from in-memory domain +objects, external ownership, and the normalized production target. It does not +invent persistence to make an ERD look complete. + +## Storage classification + +| Classification | Meaning | +|---|---| +| `persisted_runtime` | Protected-main code creates or writes this object when its adapter is enabled. | +| `in_memory` | Process state only unless projected into a generic runtime record. | +| `external_owned` | Another service or host owns the schema. | +| `accepted_target` | Reviewed target design, not the schema automatically used by protected main. | +| `active_pr` | Exists only on an open pull request. | + +## Protected-main physical objects + +| Object | Store and owner | Classification | Purpose | +|---|---|---|---| +| `agent_pool` | SQLite `_AgentPoolStore` | `persisted_runtime` | Agent ID to JSON configuration/tombstone overlay. | +| `records` | SQLite `_StateStore` | `persisted_runtime` | Generic keyed workflow/evaluation records and append-only audit/analytics payloads. | +| `records_kind_seq` | SQLite `_StateStore` | `persisted_runtime` | Kind/sequence retrieval index. | +| `provider_credentials` | Postgres `PostgresCredentialBackend` | `persisted_runtime` | pgcrypto-encrypted provider credential values. | +| `cost_attribution_dimensions` | PEP-249 `SqlLedgerStore` | `persisted_runtime` | Seven supported cost dimensions. | +| `llm_price_entries` | PEP-249 `SqlLedgerStore` | `persisted_runtime` | Created schema only; protected-main `PriceBook` does not read or write it. | +| `llm_usage_records` | PEP-249 `SqlLedgerStore` | `persisted_runtime` | Prompt-safe usage and cost facts. | +| `com_config`, `com_secrets` | `pg-llm-batch` adapter | `external_owned` | Optional external configuration/secret contract. | + +SQLite and Postgres ledger deployments use the same logical ledger columns, +subject to their driver types. No foreign keys connect any runtime table shown +below. `llm_usage_records.workflow_run_id` is informational and may contain a +workflow ID or batch job ID. The active price book lives in ConfigStore category +`llm_price_entries`, not the same-named SQL table. + +```mermaid +erDiagram + AGENT_POOL { + text agent_id PK + text payload + } + RECORDS { + integer seq PK + text kind + text key + text payload + } + PROVIDER_CREDENTIALS { + text credential_name PK + bytea encrypted_value + timestamptz updated_at + } + COST_ATTRIBUTION_DIMENSIONS { + text dimension_name PK + text dimension_label + integer dimension_order + } + LLM_PRICE_ENTRIES { + text price_entry_id PK + text provider_name + text model_name + real prompt_price_per_1k + real completion_price_per_1k + text currency_code + text updated_at + } + LLM_USAGE_RECORDS { + text usage_record_id PK + integer created_at + text workflow_run_id + text request_channel + text route_mode + text provider_name + text model_name + integer prompt_tokens + integer completion_tokens + real cost_amount + text currency_code + } +``` + +The absence of edges is intentional: protected main enforces no relational +links across these physical objects. + +## In-memory domain model + +| Domain object | Key fields | Persistence projection | +|---|---|---| +| `model_agent` (`ModelAgent`) | ID, model, base URL, credential name, tags, priority, exclusions, disabled flag | JSON in `agent_pool` only when agent DB is enabled. | +| `workflow_step` (`WorkflowStep`) | integer step ID, role, agent ID, subtask, access tuple, latency, output | Embedded in a workflow-run payload in `records` when state DB is enabled. | +| `orchestration_policy` | route P95 target, complexity threshold, verifier controls, planning mode, max steps | Snapshot inside run evidence; not a dedicated protected-main table. | +| `workflow_run` | generated ID, prompt projection, mode, answer, policy, trace, timing, usage | Memory map and optional `records(kind='workflow_run')`. | +| `evaluation_run` | generated ID, inputs, baseline/comparison measurements, workflow-run ID list | Memory map and optional `records(kind='evaluation_run')`; references are embedded JSON. | +| `audit_event` | event name, detail, time | Bounded deque and optional append-only `records(kind='audit')`. | +| `analytics_event` | event name, prompt-safe detail, time | Bounded deque and optional append-only `records(kind='analytics')`. | +| `routing_decision` | sync/batch channel and reason | Returned/embedded evidence; no dedicated table. | +| `batch_job` | job ID, status, input/result metadata | Coordinator/local maps are process-local; an external backend may own durable execution but restart loses local lookup. | +| `response_cache_entry` | request hash, timestamp, deep-copied response | Bounded in-memory TTL/LRU only. | +| `circuit_state` | failures, open-until time by agent | In-memory only. | +| `price_book_entry` | provider/model prompt and completion prices | In-memory/external ConfigStore category `llm_price_entries`; not projected to SQL `llm_price_entries`. | +| `completed_embedding_document` | batch document and usage-idempotency marker | Process-local only; lost on restart. | + +The following conceptual names make orchestration and evidence relationships +explicit without claiming that protected main creates dedicated tables: + +| Conceptual entity | Classification | Current projection or owner | +|---|---|---| +| `step_dependency` | `in_memory` | A `workflow_step.access` predecessor reference. | +| `access_grant` | `in_memory` | The validated permission to project one predecessor output into a later step. | +| `provider_credential` | `persisted_runtime` | One encrypted row in `provider_credentials`. | +| `credential_backend` | `in_memory` / `external_owned` | In-memory development adapter or injected Postgres/KV authority. | +| `cost_ledger_entry` | `persisted_runtime` | One prompt-safe `llm_usage_records` row. | +| `batch_request` | `in_memory` / `external_owned` | Local submission or request handed to `pg-llm-batch`. | +| `batch_result` | `in_memory` / `external_owned` | Qualified local/external job result. | +| `fallback_candidate` | `active_pr` | Candidate ordering in PR #94; not a protected-main stored object. | +| `check_evidence` | `external_owned` | GitHub check/run evidence, qualified by checked-out commit. | +| `release_evidence` | `accepted_target` | A provenance/SBOM/acceptance record; GitHub owns current source evidence. | + +```mermaid +erDiagram + MODEL_AGENT ||--o{ WORKFLOW_STEP : executes + ORCHESTRATION_POLICY ||--o{ WORKFLOW_RUN : governs + WORKFLOW_RUN ||--o{ WORKFLOW_STEP : contains + ACCESS_GRANT { + integer consumer_step_id + integer producer_step_id + } + WORKFLOW_STEP ||--o{ ACCESS_GRANT : consumes + WORKFLOW_STEP ||--o{ ACCESS_GRANT : produces + WORKFLOW_RUN ||--o{ AUDIT_EVENT : emits + WORKFLOW_RUN ||--o{ ANALYTICS_EVENT : emits + WORKFLOW_RUN ||--o{ USAGE_RECORD : attributes + ROUTING_DECISION ||--o| BATCH_JOB : may_submit + MODEL_AGENT ||--o| CIRCUIT_STATE : has +``` + +Each `ACCESS_GRANT` links a consumer step to an authorized producer only when +the producer is an earlier workflow step. The directional grant does not grant +bidirectional visibility. This is a conceptual ERD; these entities are not all +physical tables. + +Spend analytics, access reports, admin state, readiness resources, and +commercial packets are derived response documents, not durable entities. + +## Normalized production target + +`docs/database_design.sql` defines `agent_pool`, `orchestration_policy`, +`workflow_run`, `workflow_step`, and `audit_event`, plus retention indexes, a +safe view, and a purge function. It is `accepted_target`, not an applied +migration. + +```mermaid +erDiagram + AGENT_POOL ||--o{ WORKFLOW_STEP : executes + ORCHESTRATION_POLICY ||--o{ WORKFLOW_RUN : governs + WORKFLOW_RUN ||--o{ WORKFLOW_STEP : contains + WORKFLOW_RUN ||--o{ AUDIT_EVENT : records +``` + +Before adoption, a migration ADR must reconcile this normalized target with the +existing JSON `records` store, the current agent overlay, the cost ledger, and +host tenant/retention authority. Expand/backfill/contract and rollback evidence +are required; copying the SQL into startup code is not an accepted migration. + +## Privacy and retention implications + +- `records.payload` may contain workflow prompts and outputs. It is opt-in and + currently lacks field encryption, automatic retention pruning, tenant + partitioning, and backup policy. +- Trace payloads embed subtasks and step outputs. Audit/analytics deques are + bounded in memory, but their enabled SQLite stream rows grow without an + automatic disk-retention policy. +- `provider_credentials.encrypted_value` is encrypted at rest, but DSN, + passphrase, database authorization, rotation, and backup protection remain + deployment responsibilities. +- `llm_usage_records` intentionally excludes raw prompts and answers. +- Chat-batch result replay can create duplicate usage rows; embedding + idempotency is process-local and does not survive restart. +- The normalized target uses ciphertext, bounded previews, expiry, soft + deletion, and safe views; those controls are not retroactively claimed for + the generic SQLite store. + +## Naming exceptions + +All owned database identifiers use two-or-more-word snake_case except the +generic SQLite table `records`, which predates the repository naming contract. +That one-word object is documented technical debt. A future migration must use +a descriptive replacement such as `runtime_records`; it must not rename the +table without compatibility and rollback evidence. diff --git a/docs/INCIDENT_RUNBOOK.md b/docs/INCIDENT_RUNBOOK.md new file mode 100644 index 000000000..bec3c9056 --- /dev/null +++ b/docs/INCIDENT_RUNBOOK.md @@ -0,0 +1,158 @@ +# Incident response runbook + +**Document state:** `accepted_architecture` +**Scope:** provider, credential, privacy, persistence, cost, batch, evidence, and +release incidents owned or observed by Contextual Orchestrator + +This runbook coordinates response. It does not replace a host organization's +on-call, legal, privacy, or certification procedure. Host-owned identity, +tenancy, customer notice, and business-record decisions remain with the host. + +## Severity and authority + +| Severity | Example | Required authority | +|---|---|---| +| SEV-1 | Confirmed credential or sensitive-payload disclosure; unauthorized provider egress; destructive cross-tenant access. | Incident commander plus security/privacy owner; revoke first. | +| SEV-2 | Provider-wide outage without safe failover; corrupt durable evidence; release artifact/provenance mismatch. | Service owner plus affected dependency/release owner. | +| SEV-3 | Bounded provider degradation; dropped prompt-safe usage export; delayed batch; stale required review evidence. | Service/repository owner. | +| SEV-4 | Non-production documentation or local mock defect with no security/data impact. | Repository maintainer. | + +A model, automated reviewer, status, or local readiness endpoint cannot assume +incident-command, legal, privacy, release, or protected-merge authority. + +## Universal response + +1. **Identify:** record UTC time, reporter, affected interface, deployment, + exact source/artifact identity, policy snapshot, and observed evidence. +2. **Classify:** distinguish caller/configuration, transient upstream, + integrity/security, privacy, state, batch, cost/evidence, or release. +3. **Contain:** stop the unsafe path with the narrowest reversible control. + Revoke a credential before debugging if disclosure is plausible. +4. **Preserve:** retain bounded redacted logs, audit identities, provider + request IDs, store copies, artifact digests, and timeline. Do not copy raw + prompts, answers, credentials, or unnecessary PII into tickets. +5. **Eradicate:** perform RCA at the first failed boundary and repair the root + cause test-first. Do not weaken egress, auth, retention, coverage, review, or + release gates to restore service. +6. **Recover:** use a reviewed policy/config/artifact/store, execute the + scenario-specific checks, and observe a stable window. +7. **Reconcile:** identify lost/duplicated workflow, batch, audit, usage, or + release evidence. Mark an irrecoverable gap instead of estimating success. +8. **Close:** require protected/deployed acceptance proportional to severity, + document follow-ups/owners/dates, and update canonical docs/ADRs if the + boundary changed. + +## Credential or provider-egress incident + +1. Disable affected non-mock agents and revoke the provider credential. +2. Block the host/provider destination at the deployment boundary when safe. +3. Rotate the credential in KV through bootstrap tooling and restart any process + that could retain the old value. +4. Inspect DNS, selected address, TLS identity, proxy variables, redirect + behavior, request/response bounds, redacted exceptions, and provider audit. +5. Prove the old credential fails, the new value is read from KV, private or + mismatched destinations fail before authorization, and a permitted provider + succeeds. +6. Review whether prompts/PII crossed an unauthorized destination and invoke the + host privacy/legal process where applicable. + +Rollback means disabling the provider or returning to the last accepted +transport. It never means permitting ambient proxies, redirects, private +addresses, or environment-secret fallback. + +## Sensitive payload or PII incident + +1. Stop the affected trace, persistence, export, benchmark, or provider surface. +2. Preserve the purpose, audience, tenant, provider, residency, retention, and + authorization decision that applied. +3. Locate every authorized and unauthorized copy without broadening access. +4. Revoke credentials/tokens where needed and delete or quarantine copies under + the governing retention/legal process. +5. Verify telemetry, cost ledger, readiness packets, and default traces contain + no raw prompt/answer or credential. +6. Restore only with reviewed purpose/audience minimization, access, + encryption, provider, retention, deletion, and audit evidence. + +Blanket masking is not automatic recovery: it may destroy authorized business +meaning while leaving access and retention defects unresolved. + +## Provider outage or malformed-response incident + +1. Classify timeout, reset, throttle, unavailable/5xx, permanent 4xx, TLS, + framing, schema, or resource-bound failure. +2. Confirm bounded retry, failover, and circuit state; prevent retry storms. +3. Disable a malformed or policy-incompatible provider rather than accepting a + truncated, oversized, duplicate-key, non-finite, or incomplete response. +4. Exercise a realistic request on the recovered candidate and verify attempt, + serving-agent, validation, usage, and budget evidence. +5. Re-enable gradually under the same policy snapshot. + +## Persistence, audit, or cost-ledger incident + +1. Stop claiming durability or complete cost/audit evidence. +2. Preserve the failed SQLite/SQL/KV file/database read-only for diagnosis. +3. Determine last known good backup, write, sequence, and correlation identity. +4. Restore a validated compatible backup or initialize an explicitly new store. +5. Reconcile records, workflow IDs, batch jobs, usage export counts, drops, and + audit gaps. Unknown remains unknown. +6. Run parameter-binding, restart, migration/rollback, retention, backup, and + recovery tests before restoring the durability claim. + +## Batch dependency incident + +1. Preserve backend and client job IDs and last observed state. +2. Stop new submissions if ownership, budget, or result identity is uncertain. +3. Keep interactive route available when its dependencies are healthy. +4. Reconcile submitted, running, terminal, partial, duplicate, and unknown jobs. +5. Validate result schema, token/cost attribution, and no cross-job publication + before reopening submissions. + +Coordinator job and idempotency maps are process-local on protected main. After +a restart, do not assume an externally surviving job can be retrieved safely or +replay chat results until workflow/usage identity has been reconciled. + +## Review, check, or protected-merge incident + +1. Bind every observation to the contributor, synthetic merge, or protected + commit actually checked out. +2. Treat queued, absent, skipped-required, cancelled, failed, stale, + predecessor, author-only, status-only, rate-limited, and infrastructure-only + evidence as nonpassing. +3. Do not synthesize approval, broaden bot authority, reuse another credential, + dismiss a valid finding, or weaken protection. +4. Block only merge/release and continue non-conflicting repository work. +5. Recover through the legitimate exact-head workflow/reviewer and re-evaluate + unresolved findings and eligibility before merge. + +## Bad release, package, or migration + +1. Stop rollout/publication and preserve tag, source SHA, artifact digest, SBOM, + provenance, builder, dependency lock, and migration state. +2. Yank/deprecate according to registry policy and deploy the last accepted + artifact where safe. +3. Follow expand/backfill/contract rollback or restore the last compatible + state backup. +4. Run build/install/import, compatibility, migration/rollback, security, + reconciliation, and deployed smoke checks. +5. Publish an incident/recovery record without claiming unaffected evidence. + +## Required closure evidence + +- root-cause and systemic-control cause; +- impacted data/providers/tenants/artifacts and bounded timeline; +- containment and revocation proof; +- RED regression, minimal GREEN repair, full relevant suite; +- exact-head security/fuzz/coverage/docstring/package evidence; +- migration/rollback/reconciliation evidence where state changed; +- zero valid unresolved findings and required independent review; +- protected-main or deployed operational acceptance proportional to impact; +- updated threat model, ADR, operability, traceability, and release evidence. + +## Escalation and external evidence + +Use the repository's private vulnerability reporting path in the +[security policy](../SECURITY.md). +Production contact rosters, legal/privacy notice deadlines, buyer contacts, +cloud/provider escalation, on-call schedules, and certification-reporting +requirements are deployment-owned inputs and must be completed before a +production readiness claim. diff --git a/docs/OPERABILITY.md b/docs/OPERABILITY.md new file mode 100644 index 000000000..71f294e0e --- /dev/null +++ b/docs/OPERABILITY.md @@ -0,0 +1,143 @@ +# Operability and recovery + +**Document state:** `accepted_architecture` + +## Operating modes + +| Mode | Dependencies | Durability | +|---|---|---| +| Offline/mock | Python process and mock agent data | In-memory unless state paths are supplied. | +| Standalone provider | HTTPS model provider and credential backend | Optional SQLite state/agent files and optional SQL ledger. | +| CWL modular | Host ingress/identity plus optional KV, `pg-llm-batch`, and viewer | Each component retains its declared ownership. | + +## Startup checks + +- validate agent IDs and at least one enabled agent; +- validate bind address and explicit public-bind intent; +- validate token configuration and separation where required; +- validate configured CA bundle, state paths, KV selector, and adapter imports; +- seed provider credentials before non-mock traffic; +- report liveness separately from provider, store, batch, and review readiness. + +`GET /healthz` proves only that the process can answer. It does not prove model +credentials, provider reachability, database recovery, or release readiness. + +## Runtime signals + +- requests, rejections, active concurrency, and latency by route/conduct mode; +- selected/served agent, failover origin, transient retry, and circuit state; +- workflow step count, access exposure, verifier outcome, and trace completeness; +- provider-reported versus estimated token use and unpriced model counts; +- budget remaining/exceeded state; +- batch submitted/running/completed/failed lifecycle; +- usage export queue, stored count, failures, drops, and flush state; +- optional store availability and last successful durable write; +- exact source revision and policy/config version in operational evidence. +- execution-path label (`plain_sync`, `passthrough`, `route_stream`, or batch), + whether workflow/usage/state evidence was recorded, and why any item is absent; +- active cost authority, price source/freshness, unknown-price count, and + workflow-spend versus ledger reconciliation; +- adapter mode (`configured`, `memory_fallback`, or `heuristic_fallback`) and + process-local batch/idempotency state age. + +Raw prompts, answers, credentials, and unnecessary PII are not general metrics. + +## Initial SLI/SLO entry criteria + +Production objectives require a real deployment baseline. Repository mocks do +not establish a production SLO. Before setting targets, collect: + +- successful compatible requests divided by admitted requests; +- P50/P95/P99 latency by route/conduct/provider; +- provider failover success and retry amplification; +- trace-complete conducted-run rate; +- provider-exclusion violations (target zero); +- credential or secret exposure incidents (target zero); +- durable-state write/recovery success; +- usage export completeness and lag; +- batch completion and age by terminal state. + +## Current protected-main limitations + +- Raw passthrough and route streaming bypass coordinator ledger recording; + route streaming also bypasses durable workflow state, and failed streams may + leave no retained run. +- Workflow-derived spend/budget and the independent ledger are not synchronized. + Missing ledger price becomes zero; budget admission is pre-run and non-atomic. +- Coordinator batch/job/idempotency maps are process-local. Restart loses local + lookup and chat result replay can duplicate usage. +- Config-store and token-counter construction may silently use memory/heuristic + fallbacks. Availability does not prove the configured durable/precise mode. +- `/healthz` remains process liveness only. Static OpenAPI and runtime route + scopes are not yet generated from one registry. +- Clearfolio is a browser deep link only; this service does not own an upload or + conversion integration. + +## Incident classification + +| Class | Example | Immediate action | +|---|---|---| +| Caller/configuration | Invalid body, mode, token, missing credential, permanent 4xx. | Reject without retry; correct caller/config. | +| Transient upstream | Timeout, reset, 429, eligible 5xx. | Bounded retry/failover; observe circuit. | +| Integrity/security | Private destination, TLS/ref/hash mismatch, malformed trusted evidence. | Fail closed; do not retry as transient. | +| State | SQLite/SQL/KV unavailable or corrupt. | Stop claiming durability; isolate store and restore/replace explicitly. | +| Batch dependency | External submit/poll/retrieve unavailable. | Preserve job identity/state; keep interactive path independent. | +| Evidence/control plane | Required check/review absent, stale, or infrastructure-only. | Block only merge/release; continue safe repository work. | + +## Recovery runbooks + +### Provider outage + +1. Confirm failure class and affected agents/providers. +2. Observe bounded retry/failover and circuit state. +3. Exclude a provider only through reviewed operator policy. +4. Verify a realistic request on the recovered path. +5. Preserve the incident trace without credentials or unnecessary payloads. + +### Credential compromise + +1. Revoke at the provider and disable affected agents. +2. Rotate the KV value through bootstrap tooling. +3. Confirm old credentials fail and new credentials work. +4. Review trace/log/artifact exposure and retention. +5. Reopen the incident if any stale process can retain the old value. + +### SQLite state failure + +1. Stop treating current history as durable. +2. Preserve the failed file read-only for diagnosis. +3. Restore a validated backup or initialize an explicitly new store. +4. Reconcile workflow/evaluation/audit/analytics completeness. +5. Run restart tests before returning durability to service. + +### Cost export degradation + +1. Inspect queue/drop/store-failure telemetry without logging payloads. +2. Repair the external store or adapter. +3. Flush within a bounded time and reconcile stored counts. +4. Mark irrecoverable gaps rather than estimating missing records. + +### Bad release or migration + +1. Stop rollout and preserve exact artifact/provenance identity. +2. Follow the affected ADR's rollback or expand/backfill/contract boundary. +3. Restore a compatible state/schema version. +4. Execute package, migration, smoke, security, and data reconciliation tests. +5. Close only with protected-main/deployed evidence, not the repair PR alone. + +## Change and release operations + +- one writer per repository branch; +- exact target ref/blob rechecked before every write; +- no force push, destructive rebase, self-modifying repair workflow, or gate + weakening; +- stateful changes include forward migration, rollback, backup, and recovery; +- releases originate only from protected main after all acceptance evidence; +- version, changelog, artifacts, SBOM, provenance, and published package identity + agree. + +## External evidence still required + +Repository controls do not supply production capacity tests, hosted penetration +tests, incident-call evidence, legal/DPA acceptance, buyer signatures, or SOC 2 +and CSAP certification. Those remain explicit external inputs. diff --git a/docs/PRD.md b/docs/PRD.md new file mode 100644 index 000000000..5fa9805ec --- /dev/null +++ b/docs/PRD.md @@ -0,0 +1,187 @@ +# Product Requirements Document + +**Product:** Contextual Orchestrator
+**Document state:** `accepted_architecture`
+**Audience:** product owners, platform operators, API consumers, security and +privacy reviewers, reliability engineers, and acquisition reviewers + +## Product promise + +Contextual Orchestrator presents one provider-neutral, OpenAI-compatible model +surface while allocating work between a single-model fast path and an explicit, +auditable multi-agent workflow. It must work as a standalone service and as a +module within the ContextualWisdomLab ecosystem without transferring host +authority for identity, tenancy, durable business data, or deployment. + +The product optimizes correctness, evidence quality, reliability, control, and +cost within explicit budgets. Latency is measured and exposed, but it is not the +primary quality objective for deep orchestration. + +The product targets evidence readiness for buyer CSAP and SOC 2 diligence. It +does not claim certification, attestation, or assessor acceptance; those +remain external deployment and governance outcomes. + +## Problem + +Application teams otherwise have to implement provider selection, credentials, +retry/failover, model-group policy, context sharing, verification, cost +attribution, batch routing, and audit evidence independently. That duplication +creates inconsistent security boundaries and makes a model answer difficult to +reconstruct or govern. + +Operators need to answer five questions for every consequential run: + +1. Why was route or conduct mode selected? +2. Which model agent performed each role? +3. Which prior outputs could each step see? +4. What budget, provider, credential, and failure policy applied? +5. Which evidence is measured, estimated, absent, or external? + +## Users and jobs + +| User | Job to be done | Required outcome | +|---|---|---| +| API consumer | Replace a compatible model endpoint without learning orchestration internals. | Stable request/response semantics and explicit errors. | +| Platform operator | Configure agent pools, policy, credentials, budgets, and exclusions. | Changes are controlled, reviewable, and recoverable. | +| AI product owner | Decide when additional test-time compute improves quality. | Comparable-budget route/conduct evidence rather than anecdotes. | +| Security/privacy reviewer | Verify egress, secret, context, PII, and audit boundaries. | Least authority, purpose limitation, retention, and evidence. | +| Reliability engineer | Diagnose provider and workflow degradation. | Bounded retries, failover, circuit state, and run identity. | +| Acquisition reviewer | Determine whether claims are implemented and reproducible. | Status-qualified traceability from requirement to protected evidence. | + +## Product outcomes + +- One independently usable orchestration endpoint and one operator surface. +- Provider-neutral agent pools represented as data, not provider-specific code. +- A deterministic `route` path and an inspectable `conduct` path. +- Explicit workflow roles, subtasks, step dependencies, and access lists. +- KV-backed provider credentials; environment variables are bootstrap transport, + never request-time provider-secret authority. +- Honest cost and token evidence with estimates visibly distinguished from + provider-reported measurements. +- Sync and latency-tolerant batch routing with a standalone local backend and an + optional `pg-llm-batch` adapter. +- Fail-closed security and evidence gates without fabricated approvals, + certifications, prices, or benchmark validity. + +## Prioritized product requirements + +Release scope describes product intent, not evidence that a capability has +shipped. The capability table below supplies implementation status. + +| ID | Priority | Release scope | Accountable owner | Requirement and measurable acceptance | +|---|---|---|---|---| +| PRD-001 | P0 | current | API owner | Preserve the documented compatible subset; 100% of the versioned contract corpus passes on the release candidate. | +| PRD-002 | P0 | current | Orchestration owner | Allocate work through explicit route/conduct policy; 100% of accepted conducted runs carry one policy snapshot and ordered step evidence. | +| PRD-003 | P0 | current | Security owner | Enforce model exclusions and per-step access lists; release evidence contains zero exclusion or undeclared-access violations. | +| PRD-004 | P0 | current | Reliability owner | Bound retries, failover, circuits, concurrency, tokens, and budgets; every supported failure class terminates within its declared bound. | +| PRD-005 | P0 | current | Security owner | Resolve non-mock provider secrets from KV at final egress; credential material appears in zero logs, traces, artifacts, analytics, or model contexts. | +| PRD-006 | P0 | hardening | FinOps owner | Unify spend and ledger authorities and qualify every cost/token value as reported, measured, configured, estimated, unknown, or external; unknown price is never treated as free. | +| PRD-007 | P1 | hardening | Batch owner | Support explicit sync/batch decisions and classifiable, restart-safe, idempotent job lifecycle without coupling interactive availability to an external backend. | +| PRD-008 | P1 | production hardening | Data owner | Keep persistence opt-in and recoverable; enabled stores pass restart/restore tests and production claims wait for retention, encryption, tenancy, and backup evidence. | +| PRD-009 | P0 | next accepted stack | Security owner | Complete the DNS-pinned, proxy/redirect-safe, bounded response boundary in PR #96; acceptance requires exact-head gates and protected merge. | +| PRD-010 | P0 | every release | Release owner | Publish only one unchanged protected revision with complete functional, security, fuzz, 100% owned production statement, branch, function, and line coverage, public-docstring, package, SBOM/provenance, rollback, and independent-review evidence. | + +## Capability status + +| Capability | State | Product interpretation | +|---|---|---| +| `/v1/chat/completions`, route/conduct, trace, SSE framing | `implemented_on_protected_main` | Supported stdlib runtime surface. Conduct output is framed after synthesis; only route mode can pass through live provider tokens. | +| Configurable model agents, runtime agent-pool changes, optional SQLite overlay | `implemented_on_protected_main` | Standalone model-group management; no tenant-RBAC claim. | +| Explicit thinker/worker/verifier/synthesizer steps and access lists | `implemented_on_protected_main` | Deterministic template and validated generated-workflow seams implement the current contract. | +| Transient retry, agent failover, and per-agent circuit breaker | `implemented_on_protected_main` | Current reliability boundary; provider errors remain distinguishable from caller errors. | +| KV credential registry with in-memory and pgcrypto Postgres backends | `implemented_on_protected_main` | Provider secrets do not fall back to ambient request-time environment values. | +| Cost ledger, seven attribution dimensions, routing hints, local and `pg-llm-batch` adapters | `implemented_on_protected_main` | Two unsynchronized cost authorities exist; the SQL price table is dormant and the ledger currently treats missing price as zero. This is not cost-based provider selection and is a P0 honesty gap. | +| Optional SQLite workflow/evaluation/audit/analytics persistence | `implemented_on_protected_main` | Useful standalone durability; retention pruning and multi-tenant isolation are not complete. | +| DNS-pinned provider transport, strict bounded response parsing, and configured-Postgres fail-closed authority | `active_pr` | PR #96; source-complete security prerequisite with exact-head repository checks, but not protected-main behavior or independently approved authority until merge. | +| Local loopback MLX provider and audited model judgment | `active_pr` | PR #109; neither local-provider nor judgment behavior is protected-main product behavior until ancestry, coverage, structured review, approval, and integration gates pass. | +| Price-aware tie-breaking, administrator credential workflow, and opaque browser session | `active_pr` | PR #111 is a partial feature slice. Secure-cookie, CSRF/origin, bounded-session, restart/durability, disclosure-sink, security-base, coverage, review, and protected-integration blockers remain. | +| Fail-closed commercial release authorization | `active_pr` | PR #112 is an evidence-model prototype. Caller-supplied dictionaries are not a trusted GitHub/protected-head authority binder, so no release authority exists. | +| Equivalent-endpoint racing | `active_pr` | PR #114 is a partial immediate-race experiment. Explicit equivalence, completed-response validation, cancellation/drain, budgets, accounting, deterministic tie-breaking, delayed hedge, and comparable-budget acceptance remain incomplete. | +| Liveness/readiness split, inbound framing, and trace-authority hardening | `active_pr` | PR #121 is an open partial security slice. Duplicate framing, transfer-coding rejection, body deadlines/desynchronization, independent trace authority, dependency readiness/degraded states, security-base reconciliation, and approval remain incomplete; issues #117, #118, and #119 remain authoritative. | +| Evidence-grade NVIDIA NIM discovery and modality benchmark | `planned` | PR #90 is `superseded` closed-unmerged evidence and PR #115 is an open `superseded` scaffold; issue #86 remains the requirement authority with no accepted complete implementation PR. | +| Free-first fallback policy | `planned` | PR #94 is `superseded` closed-unmerged evidence. | +| Adaptive provider reasoning-effort control | `planned` | PR #99 is `superseded` closed-unmerged evidence. | +| Synchronous embeddings and KV-only bootstrap expansion | `planned` | PR #66 is `superseded` closed-unmerged evidence. | +| Learned routing/coordinator | `planned` | Requires a versioned evaluation set and comparable-budget proof over deterministic policy. | +| Rust/GPU mathematical or psychometric compute layer | `out_of_scope` | Required only if orchestration begins owning such arithmetic; domain services should own their scientific kernels. | + +## Functional scope + +### In scope + +- compatible chat completion and bounded streaming; +- route/conduct mode selection and policy snapshots; +- agent selection, provider exclusion, failover, and circuit state; +- workflow planning, step execution, verification, synthesis, and access lists; +- operator agent-pool, trace, audit, evaluation, cost, and readiness views; +- credential indirection and bootstrap tooling; +- honest token/cost measurement and budget enforcement; +- local and external batch adapters; +- optional standalone persistence; +- modular links to `pg-llm-batch`, Clearfolio, naruon, and other CWL hosts + through explicit interfaces. + +### Non-goals + +- training or claiming equivalence to Fugu, Conductor, or TRINITY; +- acting as an identity provider, tenant directory, DLP platform, or records + management system; +- owning another service's business data, transport, authentication, or schema; +- claiming SOC 2, CSAP, regulatory approval, production SLOs, or buyer acceptance + from repository-local checks; +- masking all PII indiscriminately. Authorized business payloads may require PII; + controls must instead combine purpose, scope, encryption, access, retention, + audit, and output minimization. Telemetry and broad traces must not copy raw + prompts or answers. + +## Test-time compute requirements + +The product must allocate a comparable call/token budget between: + +- direct single-model execution; +- route-once selection; +- bounded conducted workflows; +- reviewed fallback or deeper-recursion policies. + +Workflow stage count, recursion depth, decomposition, access lists, agent pool, +and role-specific reasoning effort are policy inputs. Evaluations report both +quality and resource use; they do not declare a deeper path better merely +because it used more calls. Fugu, Conductor, TRINITY, FrugalGPT, RouteLLM, and +newer primary work are research inputs, not compatibility claims. + +## Privacy and data-governance requirements + +- Provider credentials never enter prompts, traces, logs, or analytics. +- Raw prompts and outputs remain on the authorized execution path and are not + duplicated into usage telemetry. +- Persistence is opt-in and its operator must define purpose, access, encryption, + retention, deletion, backup, and residency. +- PII required for the authorized task is preserved inside the protected payload + path. Derived previews, broad operator views, and telemetry use minimization or + redaction appropriate to their audience. +- A host integrating this module owns end-user consent, legal basis, tenant + authorization, subject-rights workflows, and business-record retention unless + a versioned contract explicitly delegates them. + +## Success measures + +| Measure | Release target | Evidence status | +|---|---|---| +| Compatible request success | 100% pass on the versioned supported-request corpus. | Repository test evidence; production availability remains external. | +| Trace completeness | 100% of accepted conducted runs have one policy snapshot and ordered step/access evidence. | Repository and deployed sampling. | +| Provider exclusion miss rate | Zero. | Repository tests plus deployed policy audit. | +| Secret exposure | Zero credential material in logs, traces, artifacts, analytics, or model context. | Security tests and deployment review. | +| Cost evidence honesty | 100% of returned cost/token facts carry an allowed provenance; unknown price is never zero/free. | Target; protected main does not yet satisfy this across both cost authorities. | +| Comparable-budget uplift | No deeper policy is promoted without a predeclared threshold, common budget, repeated cells, and uncertainty. | Evaluation evidence; no universal uplift is claimed. | +| Recovery | Every supported provider, persistence, and batch degradation terminates in a documented state within configured bounds. | Target; process-local batch and stream-persistence gaps remain. | +| Documentation fitness | All canonical files, ADR schemas, diagrams, local links, runtime names, and data objects pass the documentation contract. | `tests/test_documentation_contract.py`. | + +## Release and acquisition acceptance + +A release candidate requires one unchanged protected head with passing +functional, security, fuzz, 100% owned production statement/branch/function/line +and public-docstring, packaging, SBOM/provenance, compatibility, and +reproducibility evidence; zero valid unresolved findings; and qualifying +independent non-author approval. Repository evidence may demonstrate controls, +but external audit, production SLO, penetration-test, DPA, buyer-signature, and +certification evidence remain explicitly external. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..78976fa19 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,67 @@ +# Authoritative documentation + +This index is the entry point for product, technical, operational, and +governance decisions. Conversation history, issue descriptions, pull-request +bodies, and implementation plans are evidence, not the product source of truth. + +## Status vocabulary + +Every capability and decision uses one of these states. + +| State | Meaning | +|---|---| +| `implemented_on_protected_main` | The behavior exists on protected `main` and has repository evidence. | +| `active_pr` | The behavior exists only on an open pull request and is not shipped. | +| `accepted_architecture` | The direction is accepted, but implementation may be partial. | +| `planned` | The work is prioritized but not accepted as implemented. | +| `research_only` | The repository contains evaluation or design evidence, not a product contract. | +| `superseded` | A newer decision or artifact replaces this one. | +| `out_of_scope` | Another system owns the behavior. | + +## Canonical set + +Each concern has exactly one authority. Volatile reviewed revisions are kept in +dated evidence appendices rather than copied into durable documents. + +| Concern | Authority | Accountable owner | State | Review trigger | +|---|---|---|---|---| +| Product intent, users, scope, outcomes | [PRD](PRD.md) | Product owner | `accepted_architecture` | Product promise, user, scope, priority, or success target changes. | +| Functional and non-functional requirements | [TRD](TRD.md) | Runtime maintainers | `accepted_architecture` | API, runtime, dependency, NFR, security, or deployment behavior changes. | +| System boundaries and component ownership | [Architecture](../ARCHITECTURE.md) | Architecture owner | `accepted_architecture` | Component, trust boundary, integration, or failure domain changes. | +| Runtime interactions and state transitions | [UML](UML.md) | Architecture owner | `accepted_architecture` | Control flow, actor, state, or deployment topology changes. | +| Persisted, in-memory, external, and target data | [ERD](ERD.md) | Data owner | `accepted_architecture` | Object, relationship, retention, migration, or ownership changes. | +| Architecture decisions | [ADR index](adr/README.md) | Affected context owner | `accepted_architecture` | A durable choice, record status, or supersession condition changes; each ADR retains its own status. | +| Requirement-to-code-to-test mapping | [Traceability](TRACEABILITY.md) | Release evidence owner | `active_pr` | Requirement status, implementation, decision, test, or operations authority changes. | +| Volatile SHA, workflow, review, and branch snapshots | [Dated evidence appendices](evidence/README.md) | Release evidence owner | `active_pr` | A new evidence collection is completed; historical appendices are not rewritten as current authority. | +| Security abuse cases and controls | [Threat model](THREAT_MODEL.md) | Security owner | `accepted_architecture` | Asset, zone, threat, control, or residual risk changes. | +| Verification strategy and evidence taxonomy | [Test strategy](TEST_STRATEGY.md) | Quality owner | `accepted_architecture` | Test layer, coverage, review, or release gate changes. | +| Operations, degraded modes, recovery, and SLO entry criteria | [Operability](OPERABILITY.md) | Service owner | `accepted_architecture` | Dependency, signal, SLO, incident, recovery, or rollout changes. | +| Incident triage, containment, recovery, and evidence preservation | [Incident runbook](INCIDENT_RUNBOOK.md) | Incident commander | `accepted_architecture` | Severity, containment, recovery, or closure authority changes. | +| Release admission, build, migration, publication, rollback, and operational acceptance | [Release guide](RELEASE_GUIDE.md) | Release owner | `accepted_architecture` | Packaging, version, provenance, deployment, migration, rollback, or release evidence changes. | +| Primary research and authoritative standards | [References](REFERENCES.md) | Architecture owner | `research_only` | A cited primary source or governing standard changes. | +| Coordinated vulnerability disclosure | [Security policy](../SECURITY.md) | Security owner | `implemented_on_protected_main` | Reporting or response process changes. | + +## Supporting evidence + +The following remain useful supporting documents but do not replace the +canonical set: + +- `conductor/` records context-driven development tracks. +- `docs/product_planning.md`, `docs/user_stories.md`, and + `docs/rest_api_design.md` preserve earlier product-design inputs. +- `docs/architecture.md` is a research mapping for Fugu, Conductor, and + TRINITY; root `ARCHITECTURE.md` is the current system authority. +- `docs/database_design.sql` is a reviewed production-target relational design, + not the schema used by every protected-main runtime mode. +- `docs/commercial_*.md` are buyer-evidence packets and readiness views, not + proof that external certifications, signatures, or production SLOs exist. +- `docs/papers/README.md` and `docs/REFERENCES.md` record research and standards. +- `docs/evidence/` records dated volatile evidence and never substitutes for a + live protected-merge decision. + +## Change discipline + +Behavior changes must update the affected canonical document, ADR, and +traceability row in the same pull request. A document may cite an active pull +request, but it must not describe that work as shipped. Volatile SHAs and run +IDs belong only in dated evidence appendices. diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md new file mode 100644 index 000000000..8a4420aa8 --- /dev/null +++ b/docs/REFERENCES.md @@ -0,0 +1,195 @@ +# Authoritative references + +**Document state:** `research_only` + +References use APA 7 style where the source provides sufficient metadata. +Research motivates hypotheses and evaluation; it does not make this repository +an implementation of a trained system or establish product superiority. + +## Orchestration, routing, and test-time compute + +Chen, L., Zaharia, M., & Zou, J. (2023). *FrugalGPT: How to use large +language models while reducing cost and improving performance* [Preprint]. +arXiv. https://doi.org/10.48550/arXiv.2305.05176 + +Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Rühle, V., +Lakshmanan, L. V. S., & Awadallah, A. H. (2024). Hybrid LLM: Cost-efficient +and quality-aware query routing. In *The Twelfth International Conference on +Learning Representations*. https://openreview.net/forum?id=02f3mUtqnM +Source-license authority: https://arxiv.org/abs/2404.14618 + +Ding, D., Mallick, A., Zhang, S., Wang, C., Madrigal, D., Garcia, M. D. C. H., +Xia, M., Lakshmanan, L. V. S., Wu, Q., & Rühle, V. (2025). *BEST-Route: +Adaptive LLM routing with test-time optimal compute* [Preprint]. arXiv. +https://doi.org/10.48550/arXiv.2506.22716 + +Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., +Keutzer, K., & Upadhyay, S. K. (2024). *RouterBench: A benchmark for multi-LLM +routing system* [Preprint]. arXiv. +https://doi.org/10.48550/arXiv.2403.12031 + +Nielsen, S., Cetin, E., Schwendeman, P., Sun, Q., Xu, J., & Tang, Y. (2026). +Learning to orchestrate agents in natural language with the Conductor. In *The +Fourteenth International Conference on Learning Representations*. +https://openreview.net/forum?id=U23A2BUKYt + +Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., +Kadous, M. W., & Stoica, I. (2024). *RouteLLM: Learning to route LLMs with +preference data* [Preprint]. arXiv. +https://doi.org/10.48550/arXiv.2406.18665 + +Sakana AI. (2026). *Sakana Fugu technical report* [Preprint]. arXiv. +https://doi.org/10.48550/arXiv.2606.21228 + +Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). *Scaling LLM test-time compute +optimally can be more effective than scaling model parameters* [Preprint]. +arXiv. https://doi.org/10.48550/arXiv.2408.03314 + +Xu, J., Sun, Q., Schwendeman, P., Nielsen, S., Cetin, E., & Tang, Y. (2026). +TRINITY: An evolved LLM coordinator. In *The Fourteenth International +Conference on Learning Representations*. +https://openreview.net/forum?id=5HaRjXai12 + +Hybrid LLM supports small/large-model difficulty routing, not a paper claim +about interactive-versus-batch channels. The repository's sync/batch policy is +a product inference and requires its own evidence. + +## HTTP, API, and observability contracts + +Bray, T. (2017). *The JavaScript Object Notation (JSON) data interchange +format* (RFC 8259). RFC Editor. https://doi.org/10.17487/RFC8259 + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC +9110). RFC Editor. https://doi.org/10.17487/RFC9110 + +Nottingham, M., Wilde, E., & Dalal, S. (2023). *Problem details for HTTP APIs* +(RFC 9457). RFC Editor. https://doi.org/10.17487/RFC9457 + +OpenAPI Initiative. (2021, February 15). *OpenAPI Specification, Version +3.1.0*. https://spec.openapis.org/oas/v3.1.0.html + +OpenAI. (n.d.-a). *API reference overview*. Retrieved August 9, 2026, from +https://developers.openai.com/api/reference/overview/ + +OpenAI. (n.d.-b). *Chat Completions*. Retrieved August 9, 2026, from +https://developers.openai.com/api/reference/chat-completions/overview/ + +OpenAI. (n.d.-c). *Error codes*. Retrieved August 9, 2026, from +https://developers.openai.com/api/docs/guides/error-codes + +OpenAI. (n.d.-d). *Streaming API responses*. Retrieved August 9, 2026, from +https://developers.openai.com/api/docs/guides/streaming-responses + +Rescorla, E. (2018). *The Transport Layer Security (TLS) protocol version 1.3* +(RFC 8446). RFC Editor. https://doi.org/10.17487/RFC8446 + +WHATWG. (n.d.). *HTML living standard: Server-sent events*. Retrieved August 9, +2026, from +https://html.spec.whatwg.org/multipage/server-sent-events.html + +World Wide Web Consortium. (2021, November 23). *Trace context*. +https://www.w3.org/TR/trace-context/ + +“OpenAI-compatible” is a tested vendor-subset label, not standards-body +certification. Admin errors may adopt RFC 9457, but compatible endpoints must +retain their explicitly tested vendor envelope. The implemented OpenAPI object +declares 3.1.0; newer specifications are not silently claimed. + +## AI, secure-development, and secrets governance + +Autio, C., Schwartz, R., Dunietz, J., Jain, S., Stanley, M., Tabassi, E., +Hall, P., & Roberts, K. (2024). *Artificial intelligence risk management +framework: Generative artificial intelligence profile* (NIST AI 600-1). +National Institute of Standards and Technology. +https://doi.org/10.6028/NIST.AI.600-1 + +Barker, E. (2020). *Recommendation for key management: Part 1—General* (NIST +SP 800-57 Part 1 Rev. 5). National Institute of Standards and Technology. +https://doi.org/10.6028/NIST.SP.800-57pt1r5 + +Booth, H., Souppaya, M., Vassilev, A., Ogata, M., Stanley, M., & Scarfone, K. +(2024). *Secure software development practices for generative AI and dual-use +foundation models: An SSDF community profile* (NIST SP 800-218A). National +Institute of Standards and Technology. +https://doi.org/10.6028/NIST.SP.800-218A + +International Organization for Standardization. (2022). *Information security, +cybersecurity and privacy protection—Information security management systems— +Requirements* (ISO/IEC Standard No. 27001:2022). +https://www.iso.org/standard/27001 + +International Organization for Standardization. (2023a). *Artificial +intelligence—Guidance on risk management* (ISO/IEC Standard No. 23894:2023). +https://www.iso.org/standard/77304.html + +International Organization for Standardization. (2023b). *Information +technology—Artificial intelligence—Management system* (ISO/IEC Standard No. +42001:2023). https://www.iso.org/standard/81230.html + +OWASP Foundation. (n.d.-a). *Secrets management cheat sheet*. Retrieved August +9, 2026, from +https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html + +OWASP Foundation. (n.d.-b). *Server side request forgery prevention cheat +sheet*. Retrieved August 9, 2026, from +https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html + +OWASP Foundation. (2025). *OWASP Top 10 for large language model applications +2025*. https://genai.owasp.org/llm-top-10/ + +Souppaya, M., Scarfone, K., & Dodson, D. (2022). *Secure software development +framework (SSDF) version 1.1: Recommendations for mitigating the risk of +software vulnerabilities* (NIST SP 800-218). National Institute of Standards +and Technology. https://doi.org/10.6028/NIST.SP.800-218 + +Tabassi, E. (2023). *Artificial intelligence risk management framework (AI RMF +1.0)* (NIST AI 100-1). National Institute of Standards and Technology. +https://doi.org/10.6028/NIST.AI.100-1 + +The pgcrypto registry is an optional encrypted database backend, not a managed +KMS claim. Production evidence still requires root-key separation, rotation, +revocation, least privilege, tenant scope, audit, backup protection, and +redaction. + +## SOC 2 and Korean CSAP evidence framing + +American Institute of Certified Public Accountants. (2022a). *2017 Trust +Services Criteria for security, availability, processing integrity, +confidentiality, and privacy (with revised points of focus—2022)*. +https://www.aicpa-cima.com/resources/download/2017-trust-services-criteria-with-revised-points-of-focus-2022 + +American Institute of Certified Public Accountants. (2022b). *2018 description +criteria for a description of a service organization's system in a SOC 2 +report (with revised implementation guidance—2022)*. +https://www.aicpa.org/resources/download/get-description-criteria-for-your-organizations-soc-2-r-report + +과학기술정보통신부. (2023). *클라우드컴퓨팅서비스 보안인증에 관한 고시* +(과학기술정보통신부고시 제2023-4호). 국가법령정보센터. +https://law.go.kr/LSW/admRulInfoP.do?admRulSeq=2100000218804 + +대한민국. (2025). *클라우드컴퓨팅 발전 및 이용자 보호에 관한 법률* (법률 +제21066호). 국가법령정보센터. +https://law.go.kr/lsInfoP.do?lsId=012266 + +한국인터넷진흥원. (2026, March 12). *클라우드서비스 보안인증(CSAP)*. +https://www.kisa.or.kr/1050603 + +SOC 2 is an independent CPA examination/report, not a certification badge. +CSAP applicability depends on the deployed cloud-service boundary and current +KISA assessment materials. Repository controls can provide readiness evidence +but cannot establish either external result. + +## Source-license authority + +arXiv. (n.d.). *License and copyright*. Retrieved August 9, 2026, from +https://info.arxiv.org/help/license/index.html + +Creative Commons. (n.d.). *Attribution-NonCommercial-NoDerivatives 4.0 +International*. Retrieved August 9, 2026, from +https://creativecommons.org/licenses/by-nc-nd/4.0/ + +An author's non-exclusive grant to arXiv does not automatically grant this +repository downstream redistribution rights. Hybrid LLM is marked CC +BY-NC-ND 4.0, which does not authorize commercial distribution. The repository +links to authoritative sources and does not vendor those PDFs without separate +permission or legal review. diff --git a/docs/RELEASE_GUIDE.md b/docs/RELEASE_GUIDE.md new file mode 100644 index 000000000..a2005f2a3 --- /dev/null +++ b/docs/RELEASE_GUIDE.md @@ -0,0 +1,156 @@ +# Release, migration, and rollback guide + +**Document state:** `accepted_architecture` + +This is the canonical operator sequence for preparing, publishing, deploying, +and accepting a Contextual Orchestrator release. It complements ADR-0011, +the test strategy, operability guide, and incident runbook. It does not claim +that an active pull request, synthetic merge, workflow status, or local build +has shipped. + +## Authority and release identity + +A releasable identity begins at one exact source commit already integrated into +protected `main`. The release owner records one immutable tuple: + +- repository and exact source commit; +- version and signed or otherwise repository-authorized tag; +- source archive and package artifact digest; +- dependency-lock digest and build-environment identity; +- CycloneDX SBOM digest and provenance statement identity; +- migration identifier and compatible rollback target, when state changes; +- exact check, security, review, and protected-main operational evidence. + +The protected commit, tag, package metadata, changelog, artifact contents, and +published version must agree. Synthetic merge commits and contributor heads may +provide scoped evidence, but neither has publication authority. + +## Admission checklist + +Before building a candidate, refetch rather than remember: + +1. protected-main tip, release target commit, tag absence, and version metadata; +2. branch protection, rulesets, required checks, security gates, and release + environment policy; +3. exact-head Tests, Security, Fuzz, package, 100% production statement/branch, + and 100% public-docstring results; +4. formal reviews, zero valid unresolved findings, and an eligible independent non-author approval + on the unchanged integrated identity; +5. lockfiles, action pins, vulnerability results, license inventory, CycloneDX + SBOM, and source-to-artifact provenance inputs; +6. compatibility, migration, backup, restore, rollback, and realistic smoke + evidence for every affected runtime mode; and +7. external inputs that are truly required, without presenting a missing buyer + signature, hosted penetration test, SLO, SOC 2, or CSAP certification as + repository success. + +Queued, pending, skipped-required, cancelled, failed, absent, stale-head, +predecessor-head, author-only, status-only, synthetic-merge-only, rate-limited, +or infrastructure-only evidence blocks only the affected gate. It is never +promoted to exact-head acceptance. + +## Build and provenance procedure + +1. Create an isolated clean build from the exact source commit. Do not reuse an + editable development environment or an artifact from a predecessor head. +2. Install from the reviewed lock and build with the repository-declared + runtime. Record toolchain and dependency-lock digests. +3. Build the source and wheel artifacts, then install each into a fresh + environment and verify import, CLI help, health, and bounded mock request + paths without materializing a live model credential. +4. Repeat the build in a second clean environment and compare normalized + contents and digests. Explain any platform-defined nondeterminism; an + unexplained mismatch is not reproducible evidence. +5. Generate the CycloneDX SBOM and provenance statement from the candidate, + then bind both to the exact source commit and artifact digest. +6. Secret-scan and malware/dependency-scan the source and artifacts. Preserve + redacted results under the repository's evidence-retention policy. +7. Produce a release manifest containing the identity tuple and links to the + exact jobs. Run IDs belong in the dated manifest, not timeless architecture. + +No build step receives `NVIDIA_NIM_API_KEY` unless a separately admitted, +bounded live-model acceptance cell actually calls a provider. Review-agent +credentials remain independent, and `COPILOT_GITHUB_TOKEN` is not a model-test +credential. + +## Migration and rollback procedure + +For any state, schema, credential-backend, or external-contract change: + +1. inventory physical persisted objects separately from in-memory, + external/host-owned, conceptual, and planned entities; +2. back up the exact affected state and prove restore before mutation; +3. use expand/backfill/contract: add compatible readers/writers, backfill with + bounded reconciliation evidence, then contract only after the rollback + window closes; +4. run upgrade, mixed-version compatibility, restart, replay/idempotency, + reconciliation, and downgrade or restore tests; +5. define the last compatible application and schema/artifact pair, maximum + rollback window, data-loss boundary, and correction owner; and +6. stop the rollout if rollback cannot preserve required authorization, + credential, audit, budget, cost, batch, or workflow evidence. + +The generic SQLite `records` object and `docs/database_design.sql` target are +not interchangeable. A migration must name the physical source and target and +must not invent persistence for an in-memory or host-owned entity. + +## Publication and deployment procedure + +1. Recheck the protected ref and every release input immediately before the + irreversible publication boundary. A changed source, rule, review, artifact, + migration, or target freezes the candidate and requires regenerated proof. +2. Create the repository-authorized tag from the admitted protected commit. +3. Publish only artifacts whose digests appear in the release manifest. Do not + rebuild between approval and publication. +4. Verify registry/package metadata and download the published artifact into a + clean environment for install/import/smoke validation. +5. Roll out progressively where deployment exists. Keep the last compatible + artifact and schema available until operational acceptance completes. +6. Record deployment target, configuration/policy digest, start/end time, + operator identity, result, and rollback decision without secrets or + unnecessary PII. + +A repository release is not a production deployment, buyer acceptance, or +certification. Each external authority records its own evidence and status. + +## Protected-main operational acceptance + +After publication or deployment, verify against the exact protected-main and +artifact identities: + +- package download, install, import, CLI, and `/healthz` behavior; +- representative mock and permitted provider-neutral request paths; +- credential bootstrap-to-KV behavior without secret retention or logging; +- provider DNS pinning, redirect/proxy rejection, response bounds, failover, + and circuit recovery where the deployed mode uses those boundaries; +- state restart, audit/evidence persistence, cost attribution, budget behavior, + and batch lifecycle for enabled backends; +- required telemetry, artifact/SBOM/provenance retrieval, and incident links; +- migration reconciliation and the continued feasibility of rollback. + +`/healthz` alone is liveness, not release acceptance. Close the release only +when the dated manifest records each applicable observation or an explicitly +owned external gap. + +## Abort and recovery conditions + +Abort publication or rollout on any identity mismatch, new valid finding, +missing required approval/check, vulnerability, secret exposure, corrupt or +partial artifact, failed restore/migration, unexplained reproducibility drift, +provider trust-boundary regression, or operational smoke failure. + +Preserve the failed source, tag, artifact digest, SBOM, provenance, migration, +and redacted logs. Stop further rollout, restore the last compatible +artifact/state pair, reconcile writes and evidence, and follow the incident +runbook. Recovery creates a new candidate from a new exact protected identity; +it never edits or republishes an already approved artifact under the same +version. + +## Related authority + +- [ADR-0011: Release coverage and provenance](adr/0011-release-coverage-and-provenance.md) +- [Test strategy](TEST_STRATEGY.md) +- [Operability](OPERABILITY.md) +- [Incident runbook](INCIDENT_RUNBOOK.md) +- [Threat model](THREAT_MODEL.md) +- [Traceability](TRACEABILITY.md) diff --git a/docs/TEST_STRATEGY.md b/docs/TEST_STRATEGY.md new file mode 100644 index 000000000..ca2271500 --- /dev/null +++ b/docs/TEST_STRATEGY.md @@ -0,0 +1,81 @@ +# Test and evidence strategy + +**Document state:** `accepted_architecture` + +## Test layers + +| Layer | Purpose | Representative evidence | +|---|---|---| +| Unit and contract | Domain invariants, parsers, policies, cost arithmetic, naming, API shapes. | `tests/test_*.py` | +| Realistic integration | Local HTTP, SSE, provider failure, SQLite restart, batch lifecycle, admin/API behavior. | provider, server, persistence, batch, and admin tests | +| Property/fuzz | Untrusted request, config, redaction, and orchestration seams. | Hypothesis and bounded Atheris workflows | +| Security | CodeQL, dependency review, pip-audit, SBOM, Trivy, OSV, Scorecard, Semgrep under repository/central ownership. | Exact workflow jobs and security results | +| Research contract | Turn primary-paper concepts into observable route/conduct, roles, and access-list behavior. | `tests/test_paper_contracts.py` | +| Documentation fitness | Keep canonical documents, statuses, supported Mermaid syntax, ADRs, runtime class/method references, selected control-flow edges, and data ownership coherent. | `tests/test_documentation_contract.py` | +| Package/release | Build, install, import isolation, provenance, reproducibility, and release artifact identity. | Required before release; not inferred from unit tests | +| Live-model evaluation | Comparable-budget route/conduct/fallback/effort cells with provenance and uncertainty. | Uses bounded `NVIDIA_NIM_API_KEY`; never required for offline unit tests | + +## Coverage contract + +- Release evidence requires 100% owned production statement and branch coverage + and 100% public-docstring coverage. +- Exclusions are allowed only for genuinely unreachable platform or optional + integration paths with separate executable evidence, never to hide ordinary + behavior. +- A coverage test that exposes a real 4xx/5xx or state-transition defect becomes + a product-defect RCA; the test is not weakened to preserve a percentage. +- Every valid defect follows observable RED, minimal root-cause fix, focused + GREEN, full suite, then exact-head CI/security/fuzz proof. + +## Realism requirements + +- Provider tests distinguish caller errors, transient errors, retry exhaustion, + failover, circuit opening, and recovery. +- Streaming tests use valid and malformed SSE framing and final `[DONE]` state. +- Persistence tests create a new process object over the same store and verify + restart semantics, parameter binding, bounds, and failure behavior. +- Cost tests distinguish reported/estimated tokens, configured/unknown prices, + attribution, and export degradation. +- Batch tests cover submit, poll, retrieve, partial/malformed results, token + splitting, and external-backend failure. +- Privacy tests use credential-shaped and PII-bearing fixtures and assert + audience-appropriate preservation or minimization. +- Live orchestration comparisons use fixed tasks, common call/token budgets, + versioned scorers, repeated cells, uncertainty, and full assignment evidence. + +## Exact-head evidence taxonomy + +| Evidence | May prove | +|---|---| +| Contributor-head job that explicitly checks out that SHA | Repository-local behavior at that head. | +| Synthetic merge job | Integration behavior for that exact synthetic tree only. | +| Commit status | The named status producer's claim only. | +| Automated review | Findings/verdict from that identity and head only. | +| Independent human approval | Repository review governance if the reviewer is eligible and head remains current. | +| Protected-main run | Operational behavior of the integrated protected revision. | + +Queued, pending, skipped-required, cancelled, absent, failed, predecessor-head, +stale-base, author-only, status-only, rate-limited, and infrastructure-only +states are not success. + +## Model-test credential boundary + +Offline tests use `mock://` agents and no provider secret. A deliberately live +model evaluation receives `NVIDIA_NIM_API_KEY` only in the step performing the +bounded call, after deterministic admission checks. `COPILOT_GITHUB_TOKEN` is +not a model-development credential. Review-agent credentials remain separate. + +## Release acceptance + +One unchanged integrated head must have: + +1. full functional, realistic integration, fuzz, and security evidence; +2. 100% coverage/docstrings under the repository contract; +3. package build/install/import and compatibility proof; +4. SBOM, provenance, and reproducibility evidence; +5. current reviews, zero valid unresolved findings, and independent approval; +6. migration/rollback and operator acceptance for affected state; +7. protected-main smoke or scheduled evidence before incident closure. + +No version bump or publication occurs from an unmerged branch or synthetic +tree. diff --git a/docs/THREAT_MODEL.md b/docs/THREAT_MODEL.md new file mode 100644 index 000000000..a53f25d7d --- /dev/null +++ b/docs/THREAT_MODEL.md @@ -0,0 +1,91 @@ +# Threat model + +**Document state:** `accepted_architecture`
+**Scope:** standalone runtime, optional adapters, and CWL composition boundary + +## Assets + +- provider credentials and KV bootstrap material; +- authorized prompts, PII-bearing business content, model outputs, and traces; +- policy, agent-pool, provider-exclusion, budget, and price configuration; +- workflow, access-list, audit, analytics, benchmark, and release evidence; +- state, credential, cost-ledger, and external batch stores; +- reviewer, check, status, merge, and release authority. + +## Trust zones + +1. Untrusted caller and payload. +2. Authenticated inference caller. +3. Privileged operator/admin. +4. Orchestration process and its memory. +5. Credential and state stores. +6. External model provider and network. +7. Optional batch/viewer/CWL dependencies. +8. GitHub CI, automated reviewers, human reviewers, and protected merge. + +## Threats and controls + +| Threat | Abuse path | Current control | Residual gap/status | +|---|---|---|---| +| Authentication bypass | Public bind or shared token exposes admin or inference. | Bearer checks, split admin/inference tokens, explicit public-bind flag, request bounds. | Production identity, rotation, tenant RBAC, and gateway policy are host-owned/`planned`. | +| SSRF and DNS rebinding | Provider base URL resolves to internal or changed destination. | Protected main rejects non-HTTPS/non-global addresses and supports allowlists. | Full DNS pinning, proxy/redirect rejection, and strict transport lifetime are `active_pr` #96. | +| Credential exfiltration | Secret leaks into model context, redirects, logs, errors, traces, or artifacts. | KV lookup, no ambient runtime fallback, redaction, prompt-safe ledger. | Rotation and audience-bound secret distribution require deployment evidence. | +| Malicious provider response | Oversized, malformed, duplicate-key, non-finite, or deceptive response exhausts or contaminates runtime. | Provider URL validation, a socket timeout, a request-side `max_tokens` hint, and expected-shape validation seams. | There is no enforced provider-response byte cap or cumulative SSE cap; strict bounded framing/JSON/SSE is `active_pr` #96. | +| Prompt injection across agents | One model output instructs later roles or steals hidden context. | Access lists limit visibility; roles and subtasks are explicit. | Content remains untrusted; tool authority and sanitization belong to each integrating tool/host. | +| Excessive context disclosure | Generated workflow exposes unrelated prior outputs. | Plan validation, explicit step access tuple, trace inspection. | Tenant/purpose authorization is host-owned; role-effort control is `active_pr` #99. | +| Denial of wallet/service | Large prompts, deep workflows, retries, concurrency, provider output, or batch fan-out exhaust resources. | Request-body, workflow-step, retry, rate, concurrency, cache, and workflow-budget controls; provider calls carry an output-token hint. | Hard upstream response bounds, distributed quotas, and provider-level reservation are incomplete. | +| Cost falsification | Estimated tokens or missing price appears as measured spend/free routing. | Spend analytics qualifies unknown price, but the separate ledger defaults missing price to zero and its SQL price table is dormant. | Unify cost authority, make unknown non-zero/non-free, and reconcile budget/ledger before cost-routing claims. | +| Persistence disclosure | SQLite JSON payload or backup exposes raw prompts/outputs. | Persistence is opt-in; SQL parameters prevent injection; normalized encrypted target is documented. | Generic state encryption, retention pruning, tenancy, and backup controls are not complete. | +| PII loss through blanket masking | Required business identifiers are destroyed or become unusable. | Payload path may retain authorized data; broad telemetry omits prompt/output. | Audience- and purpose-specific field policy needs host integration; masking alone is rejected. | +| PII overexposure | Full traces or persisted payloads reach a broader audience than the inference request. | Trace is not exposed by default; admin/inference tokens can be split; redaction/minimization. | Protected main has no dedicated trace scope: an inference caller can opt into a chat trace. Fine-grained tenant/purpose RBAC and subject-rights workflow are host-owned. | +| SQL/config injection | Attacker-controlled kind/key/payload alters schema or query. | Bound SQL parameters and naming validation. | Generic JSON payload schema evolution needs migration controls. | +| Cache data crossover | Cached response is reused for a different caller or policy. | Exact request/mode key, disabled by default, bounded TTL/LRU, deep copies. | Multi-tenant cache partitioning is not implemented; hosts should keep cache disabled absent an authority key. | +| External batch confused deputy | Caller submits/retrieves another job or replays results for duplicate accounting. | Injected backend contract, request validation, local standalone backend. | Tenant ownership is adapter/host-owned; coordinator handles and idempotency are process-local and lost on restart. | +| Evidence-path bypass | Passthrough or route streaming returns an answer without the workflow, ledger, budget, or durable record expected by operators. | Mode-specific analytics/traces and explicit documentation. | Protected main still bypasses coordinator usage for passthrough/streaming and state persistence for route streaming. | +| Silent adapter downgrade | Config or token-count adapter failure falls back to memory/heuristic behavior without sufficient operator authority. | Standalone fallback preserves availability. | Degraded mode must be surfaced and excluded from durable/precise evidence claims. | +| Supply-chain compromise | Mutable action/package or generated artifact executes attacker code. | Immutable action pins, hash locks, CodeQL, dependency audit, SBOM, central scanners. | External runner and provider integrity require operational evidence. | +| Evidence/reviewer spoofing | Status or model comment is treated as independent approval or exact-head success. | Explicit evidence taxonomy and branch governance. | Live rulesets and eligible human capacity remain external governance dependencies. | +| Stale-base merge | Check/review applies to predecessor head or synthetic merge. | Exact contributor-head workflows and live-base reconciliation policy. | Central control-plane reliability must be proven on protected main. | + +## Misuse cases + +### Authorized PII request + +An authenticated host sends a customer email requiring names and account data. +The orchestrator may pass the minimum necessary content to an approved model +under the host's purpose policy. Usage telemetry receives identifiers and token +counts, not the message body. Protected main does not enforce a separate trace +permission: an inference-scoped chat caller may opt into orchestration trace +output, while admin authority can retrieve persisted workflow records. The host +must therefore restrict those tokens/endpoints to the same authorized purpose; +dedicated tenant/purpose trace RBAC remains planned. + +### Hostile provider configuration + +An operator attempts to configure a provider at loopback, a private network, or +an allowlist-excluded host. Execution fails before sending a credential. The +active #96 transport strengthens this by retaining validation-time address pins +through connection establishment. + +### Compromised worker output + +A worker emits instructions to reveal another step's context. The verifier and +synthesizer receive only their declared access-list inputs. They must treat +worker text as untrusted data; no tool or credential authority follows from the +text. + +## Security acceptance + +- realistic auth, SSRF, redirect, response-bound, malformed JSON/SSE, secret + leakage, PII, SQL, cache, concurrency, retry, batch, and evidence tests; +- 100% owned production statement and branch coverage without excluding real + behavior; +- current exact-head CodeQL, dependency, supply-chain, fuzz, and required + organization scans; +- zero valid unresolved findings and qualifying independent approval; +- deployment-specific penetration, access, retention, backup, and incident + evidence before production claims. + +The repository is designed toward NIST AI RMF, NIST SSDF, ISO/IEC 27001, +ISO/IEC 23894, ISO/IEC 42001, CSAP, and SOC 2 evidence needs. It does not claim +certification or attestation. See `REFERENCES.md`. diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md new file mode 100644 index 000000000..d73a94a0d --- /dev/null +++ b/docs/TRACEABILITY.md @@ -0,0 +1,99 @@ +# Requirement traceability + +**Document state:** `active_pr`
+**Canonical role:** durable requirement-to-implementation-to-decision-to-test mapping + +This document records stable authority relationships. Volatile commit SHAs, +workflow IDs, review snapshots, and current branch state belong only in the +[dated evidence appendices](evidence/README.md). A pull request, issue, or +dated appendix is evidence and never becomes shipped authority until protected +integration and operational acceptance succeed. + +## Requirement-to-implementation matrix + +| Requirement | Product state | Implementation authority | Decision and operational authority | Test authority | +|---|---|---|---|---| +| PRD-001 / FR-001 compatible chat surface | `implemented_on_protected_main` | `server.py`, `orchestrator.py` | PRD, TRD, ADR-0002, operability | API and passthrough tests | +| PRD-002 / FR-002 route/conduct allocation | `implemented_on_protected_main` | `TaskOrchestrator.complete` | ADR-0001, UML | paper and optimizer tests | +| PRD-003 / FR-003/004 workflow and access control | `implemented_on_protected_main` | `WorkflowStep`, conduct planner | ADR-0003, UML | workflow and access tests | +| PRD-004 / FR-005 reliability and failover | `implemented_on_protected_main` | `ModelClient`, orchestrator circuit state | Architecture, threat model, incident runbook | provider reliability tests | +| PRD-005 / FR-006 KV credentials | `implemented_on_protected_main` | `credentials.py`, `kv_config.py`, CLI | ADR-0004, UML, threat model | KV credential tests | +| PRD-006 / FR-007 cost attribution | `implemented_on_protected_main` with honesty gaps | `orchestrator.py`, `cost_ledger.py`, `cost_router.py` | ADR-0006, ERD, operability | ledger reconciliation and unknown-price tests | +| PRD-007 / FR-008 sync and batch | `implemented_on_protected_main` with restart gaps | `batch_routing.py`, `cost_router.py` | ADR-0005, UML, operability | batch restart and replay tests | +| PRD-008 / FR-009 optional persistence | `implemented_on_protected_main` | state, agent-pool, SQL, and KV adapters | ADR-0008, ERD, release guide | persistence and migration tests | +| FR-011 route registry parity | `accepted_architecture` | dispatcher and generated contract | TRD, test strategy | shared-registry parity tests | +| FR-012 execution-path evidence parity | `accepted_architecture` | route, conduct, passthrough, streaming, and batch paths | Architecture, UML, threat model | execution-mode matrix | +| PRD-006 / FR-013 cost authority | `accepted_architecture` | price and usage authorities | ADR-0006, ERD | reconciliation and non-free unknown-price tests | +| PRD-007 / FR-014 durable batch identity | `accepted_architecture` | job and idempotency stores | ADR-0005, operability | restart and replay tests | +| PRD-009 / SEC-002 provider transport and configured-KV trust | `active_pr` | PR #96 | ADR-0002, ADR-0015, threat model | PR-bound evidence until protected merge | +| Local loopback MLX provider and audited model judgment | `active_pr` | PR #109 independently targets protected main | PR #109 planning ADR, PRD, TRD | PR-bound evidence until protected merge | +| Price-aware tie-breaking and administrator credential workflow | `active_pr` | PR #111 is a partial feature slice on protected-main ancestry | PR #111 contracts and issue #116 | remaining session, security, coverage, review, and protected-integration evidence | +| Fail-closed commercial release authorization | `active_pr` | PR #112 is an evidence-model prototype, not a trusted authority binder | ADR-0010, ADR-0011, release guide | spoofing, ruleset, review-identity, and protected-integration matrix remains incomplete | +| Equivalent-endpoint racing | `active_pr` | PR #114 is a partial immediate-race experiment | issue #102, architecture, operability, threat model | equivalence, validation, cancellation, accounting, and ablation matrix remains incomplete | +| Liveness/readiness, inbound framing, and trace authority | `active_pr` | PR #121 is an open partial security slice | issues #117, #118, and #119 remain requirement authority | duplicate framing, deadlines, independent trace authority, readiness degradation, and protected integration remain incomplete | +| Evidence-grade NVIDIA NIM benchmark | `planned` | PR #115 is an open `superseded` scaffold; issue #86 owns replacement (predecessor PR #90 evidence recorded below) | ADR-0006 | replacement benchmark evidence | +| Free-first fallback | `planned` | PR #94 is `superseded` closed-unmerged evidence | ADR-0007 | replacement implementation requires fresh protected-line evidence | +| Adaptive reasoning effort | `planned` | PR #99 is `superseded` closed-unmerged evidence | ADR-0003 | comparable-budget replacement tests | +| Synchronous embeddings and KV-only bootstrap | `planned` | PR #66 is `superseded` closed-unmerged evidence | PRD and TRD | replacement contract evidence | +| PRD-010 independent review and release | `accepted_architecture` | repository rules, workflows, and human governance | ADR-0010, ADR-0011, ADR-0016, release guide | exact-head and protected-main evidence | +| Purpose-bound PII handling | `accepted_architecture` | host and runtime audience boundaries | ADR-0009, threat model | privacy, telemetry, and trace tests | + +## Authority flow + +```mermaid +flowchart TB + Requirement["PRD and TRD requirement"] --> Decision["Status-bearing ADR"] + Decision --> Runtime["Runtime or active PR implementation"] + Runtime --> Test["Deterministic test authority"] + Test --> Operations["Operability and incident authority"] + Operations --> Evidence["Dated evidence appendix"] + Evidence --> Gate["Protected release gate"] + Gate -->|accepted unchanged head| ProtectedMain["Protected main"] + Gate -->|absent, stale, failed, or synthetic evidence| Blocked["Blocked"] +``` + +## Active stack relationships + +PR #96 supersedes closed-unmerged PR #76. PR #82 is `superseded` +closed-unmerged bootstrap evidence; only its unique intent may be rebuilt after +PR #96 reaches an accepted protected result. PR #105 carries this canonical +documentation graph. PR #104 was merged into this documentation stack and is +not protected-main authority until PR #105 reaches protected main. + +PR #109 remains an independent `active_pr` local-provider/judgment slice. PR +#111, PR #112, PR #114, and PR #121 are independent `active_pr` partial or +prototype slices; none inherits PR #96 authority or closes its owning issue. +PR #115 is an open scaffold classified `superseded`, not active implementation +authority. PR #113 and PR #120 are `superseded` closed-unmerged duplicate +documentation replays; their accepted unique intent is retained in this PR #105 +graph. No predecessor, author-only, status-only, queued, or synthetic-merge +evidence transfers between these branches. + +## Open product backlog relationships + +- Issue #95 closes only after PR #96 reaches protected main and operational + acceptance succeeds. +- Issue #103 has a partial prototype in active PR #112. The trusted GitHub and + protected-main authority binder remains incomplete, so the issue stays open. +- Issue #102 has a partial immediate-race experiment in active PR #114. Explicit + endpoint equivalence, completed-response validation, cancellation/drain, + accounting, deterministic tie-breaking, and comparable-budget acceptance + remain incomplete. +- Issue #86 owns evidence-grade NIM model discovery and cost-quality evaluation. + Predecessor PR #90 is `superseded` closed-unmerged evidence. + PR #115 is an open scaffold classified `superseded`, not active + implementation authority, because it does not satisfy the accepted + security, benchmark, and evidence contract. +- Issues #117, #118, and #119 remain open requirement authorities. Active PR + #121 supplies partial implementation evidence only and does not close them. +- Issue #116 remains open. Active PR #111 proves opaque session separation, + expiry, and logout, but Secure-cookie, CSRF/origin, bounded-session, + restart/durability, disclosure-sink, security-base, and approval acceptance + remain incomplete. + +## Documentation maintenance rule + +Any mapped behavior change must update the relevant PRD/TRD status, ADR, +diagram, data ownership, test authority, operability path, and this matrix in +the same reviewed change. Evidence-only changes update a dated appendix rather +than embedding volatile identifiers in canonical documents. diff --git a/docs/TRD.md b/docs/TRD.md new file mode 100644 index 000000000..4422bde55 --- /dev/null +++ b/docs/TRD.md @@ -0,0 +1,291 @@ +# Technical Requirements Document + +**Document state:** `accepted_architecture`
+**Implementation baseline:** protected `main`; volatile evidence is recorded in +the dated audit in `TRACEABILITY.md`
+**Package baseline:** version `0.1.0`; Python `>=3.10` + +## System context + +Contextual Orchestrator is a Python service and library. The protected-main +runtime has no mandatory third-party runtime dependency for its standalone +path. PostgreSQL and `pg-llm-batch` are optional adapters. FastAPI, SQLAlchemy, +and Alembic are dependency/design targets but are not used by the shipped +stdlib HTTP runtime. + +The public service uses `contextual_orchestrator.server`. For ordinary chat, +`CostRoutingCoordinator` first owns sync-versus-batch choice, token counting, +and the independent cost ledger; the sync path then invokes +`TaskOrchestrator`, which owns route-versus-conduct policy, model/role +selection, execution, traces, evaluation, audit, caches, workflow-derived +budget, and optional SQLite state. `ModelClient` owns provider calls. Raw +passthrough and route-streaming paths bypass parts of that composition and are +explicit gaps below. Agent definitions remain configuration data. + +## Requirement identifiers + +### Functional requirements + +| ID | Requirement | Current state | Verification authority | +|---|---|---|---| +| FR-001 | Accept the documented OpenAI-compatible chat-completion subset. | `implemented_on_protected_main` | `tests/test_openai_passthrough.py`, `tests/test_api_contract.py` | +| FR-002 | Select `route` or `conduct` deterministically from caller mode and policy. | `implemented_on_protected_main` | `tests/test_paper_contracts.py`, `tests/test_optimizer.py` | +| FR-003 | Represent conducted work as ordered steps with role, agent, subtask, access list, latency, and output. | `implemented_on_protected_main` | `tests/test_generated_workflow.py`, `tests/test_paper_contracts.py` | +| FR-004 | Restrict each step context to its declared access list. | `implemented_on_protected_main` | `tests/test_paper_contracts.py` | +| FR-005 | Retry transient provider failures, fail over to an eligible agent, and open/close per-agent circuit state. | `implemented_on_protected_main` | `tests/test_provider_reliability.py` | +| FR-006 | Resolve provider credentials from the configured KV backend. | `implemented_on_protected_main` | `tests/test_kv_credentials.py` | +| FR-007 | Attribute usage across account, service, upstream API, model, team, group, and company. | `implemented_on_protected_main` | `tests/test_cost_ledger.py` | +| FR-008 | Route latency-tolerant work to local or injected batch backends and expose job lifecycle. | `implemented_on_protected_main` | `tests/test_batch_routing.py`, `tests/test_cost_review_server.py` | +| FR-009 | Persist workflow/evaluation/audit/analytics and agent overlays only when explicitly configured. | `implemented_on_protected_main` | `tests/test_persistence.py`, `tests/test_agent_pool_db.py` | +| FR-010 | Expose operator views without returning full traces to untrusted callers by default. | `implemented_on_protected_main` | `tests/test_security_hardening.py`, `tests/test_admin_contract.py` | +| FR-011 | Generate dispatch, OpenAPI, scopes, and endpoint documentation from one route registry. | `accepted_architecture` | parity test required; protected main is incomplete | +| FR-012 | Record or explicitly qualify workflow, persistence, budget, and usage evidence consistently across plain, passthrough, streaming, and batch paths. | `accepted_architecture` | mode-by-mode integration matrix required | +| FR-013 | Use one price/cost authority in which unknown price remains unknown and cost-based selection is claimed only when invoked. | `accepted_architecture` | ledger/spend reconciliation and unknown-price tests required | +| FR-014 | Preserve restart-safe, idempotent batch job identity and usage recording. | `accepted_architecture` | restart/retrieval/replay tests required | +| FR-015 | Support a bounded local loopback MLX provider and fail-closed audited model judgment without granting the local transport broader credential or egress authority. | `active_pr` | PR #109 tests and planning ADR; no protected-main authority until merge | +| FR-016 | Separate unauthenticated process liveness from bounded, authenticated dependency readiness and deterministic degraded states. | `active_pr` | PR #121 is a partial implementation; real dependency probes, deadlines, 503/degraded semantics, and protected integration remain incomplete | +| FR-017 | Reject ambiguous, duplicate, transfer-coded, oversized, truncated, or slow inbound request framing before unbounded body consumption and close unsafe connections. | `active_pr` | PR #121 is a partial implementation; duplicate-length, transfer-coding, deadline, connection-close, socket, and fuzz acceptance remain incomplete | +| FR-018 | Authorize orchestration-trace disclosure independently from ordinary inference/admin access across every trace-bearing surface. | `active_pr` | PR #121 is a partial implementation; purpose, tenant, resource, lifetime, revocation, compatibility, and complete-surface acceptance remain incomplete | + +### Quality and safety requirements + +| ID | Requirement | +|---|---| +| NFR-001 | The standalone mock path is deterministic, offline, and installable without provider credentials. | +| NFR-002 | Public APIs and database objects use two-or-more-word snake_case except external-standard fields and documented paper roles. | +| NFR-003 | Owned production code maintains 100% statement, branch, function, and line coverage where tooling exposes each dimension, plus beginner-readable public docstrings, before release. | +| NFR-004 | Concurrency bounds, body bounds, output-token bounds, timeouts, retries, cache limits, and budgets are explicit. | +| NFR-005 | A degraded optional store or integration cannot silently falsify success or cost evidence. | +| NFR-006 | Every measurement identifies its source as reported, measured, configured, estimated, unknown, or external. | +| NFR-007 | The module works independently and does not require a CWL control plane for basic operation. | + +### Security and privacy requirements + +| ID | Requirement | +|---|---| +| SEC-001 | Non-mock provider credentials are retrieved from KV at the final execution path and never fall back to ambient request-time environment values. | +| SEC-002 | Non-mock provider URLs use HTTPS and reject non-global destinations; DNS pinning, redirect/proxy rejection, and strict response bounds remain `active_pr` until PR #96 merges. | +| SEC-003 | Caller and admin bearer authority are separable; public bind requires an explicit operator choice. | +| SEC-004 | Raw secrets never enter analytics, exception text, stored evidence manifests, or model-visible context. | +| SEC-005 | PII handling is purpose- and audience-bound. Blanket masking is not a substitute for authorization, encryption, retention, deletion, or audit. | +| SEC-006 | Untrusted JSON, SSE, agent configuration, and redaction inputs have validation and fuzz seams. | +| SEC-007 | Checks, statuses, reviews, and merge authority are distinct evidence types; none may impersonate another. | +| SEC-008 | Browser-admin sessions are cryptographically and semantically distinct from long-lived bearer credentials, bounded, revocable, origin/CSRF-controlled, Secure on HTTPS, and explicit about restart/durability semantics. PR #111 is a partial `active_pr` implementation. | +| SEC-009 | Trace disclosure requires an independent purpose-bound authority rather than inference or broad admin scope alone. PR #121 is a partial `active_pr` implementation. | + +## Interfaces + +### Inference and batch + +- `POST /v1/chat/completions` +- `POST /v1/responses` +- `POST /v1/batch/embeddings` +- `GET /v1/batch/embeddings/{batch_id}` +- `POST /api/v1/batch_routing_jobs` +- `GET /api/v1/batch_routing_jobs/{batch_job_id}` +- `POST /api/v1/batch_routing_jobs/{batch_job_id}/results` + +### Operator and evidence + +- `/admin` and `/admin/state` +- `/api/v1/agent_pools/{agent_pool_id}/worker_agents` +- `/api/v1/workflow_runs` and individual workflow records +- `/api/v1/evaluation_runs` +- `/api/v1/access_reports/{workflow_run_id}` +- cost, usage, analytics, readiness, and buyer-evidence resources defined in + `contextual_orchestrator/api_contract.py` +- `GET /healthz` on protected main combines liveness with internal detail and + is not an accepted readiness contract; active PR #121 partially narrows it +- `GET /readyz` exists only on active PR #121 and is not accepted until it + proves bounded dependency health and deterministic degraded states + +Current protected-main scopes are asymmetric: `/healthz` and `/openapi.json` are +unauthenticated; chat, Responses, embedding batch, chat-batch submission and +result upload, workflow creation, and evaluation creation use inference +authority; most operator routes use admin authority. In protected main, chat +batch polling by `GET` falls through the admin gate even though submit/results +are inference-scoped. Treat that as an explicit compatibility/authorization +decision before changing it. + +There is no dedicated trace scope on protected main. A caller with inference +authority can set `include_orchestration_trace: true` on chat requests; separate +tenant/purpose trace authority therefore belongs at the host/gateway boundary +until runtime RBAC is added. Active PR #121 partially removes inference-only +trace disclosure on selected paths, but broad admin authority still acts as +trace authority and several trace-bearing surfaces and malformed-input cases do +not share a complete independent policy. + +The dispatcher in `server.py` is the current delivery truth. `OPENAPI_SPEC` in +`api_contract.py` describes only a resource-oriented subset and omits +implemented chat, Responses, health, spend, admin, and agent create/delete +routes. The intended architecture is one shared route registry that generates +dispatch, scopes, OpenAPI, and endpoint documentation. + +“OpenAI-compatible” means only the versioned subset tested here: request fields, +sync response, Chat Completions `delta`/`[DONE]` streaming, auth header, model +mapping, error behavior, and unknown-field policy. It is not standards-body +conformance. Responses API typed events are not interchangeable with Chat +Completions SSE chunks, and admin/control-plane errors need not use the vendor +envelope. + +## Execution requirements + +### Route + +1. Authenticate and validate the request. +2. Enforce budget and concurrency policy. +3. Select one eligible model agent from current pool data. +4. Resolve its credential from KV for non-mock execution. +5. Execute with bounded transient retries, failover, and circuit policy. +6. On the ordinary synchronous coordinator path, record usage, audit, and trace + evidence with source qualifications. +7. Return a compatible response. Route streaming may relay provider deltas, + but protected main bypasses the cost coordinator and does not persist its + workflow run to `_StateStore`; no cross-agent failover occurs after bytes + are emitted. + +Requests containing tools, functions, or `response_format` use raw passthrough +on protected main. They record analytics but no workflow run or cost-ledger row. +That distinction must remain visible until FR-012 is satisfied. + +### Conduct + +1. Classify or accept explicit conduct mode. +2. Build a bounded template or validated generated plan. +3. Execute ordered roles while including only declared predecessor outputs. +4. Verify and synthesize under the policy snapshot. +5. Persist or retain evidence according to configured runtime mode. +6. Return the answer; any stream is post-synthesis framing, not live upstream + token pass-through. + +## Persistence requirements + +Protected main has four distinct persistence boundaries: + +1. `_StateStore`: optional SQLite `records` table for keyed workflow/evaluation + records and append-only audit/analytics streams. +2. `_AgentPoolStore`: optional SQLite `agent_pool` overlay with JSON payloads. +3. `PostgresCredentialBackend`: optional pgcrypto-encrypted + `provider_credentials` registry. +4. `SqlLedgerStore`: PEP-249 tables `cost_attribution_dimensions`, + `llm_price_entries`, and `llm_usage_records` on SQLite or Postgres. + +`PriceBook` currently reads ConfigStore category `llm_price_entries`; it does +not read or write the SQL `llm_price_entries` table. Protected main also has two +unsynchronized cost authorities: workflow-derived spend analytics/budget and +the independent ledger. The default ledger price book is empty and missing +prices become `0.0`, while spend analytics reports missing price as unknown. +`cheapest_upstream()` is not invoked by orchestration, so protected main does +not ship price-based provider selection. + +Coordinator batch handles and local/external request-result mappings are +process-local. Restart loses lookup authority even if an external job survives; +chat-batch result replay can duplicate usage rows. Embedding results have only a +process-local idempotency guard. + +`docs/database_design.sql` is a normalized production target. It must not be +described as the schema automatically created by the standalone runtime. +External `pg-llm-batch` configuration and secret tables are owned by that +adapter/service. + +## Credential requirements + +- `CONTEXTUAL_ORCHESTRATOR_KV_BACKEND`, KV DSN, and KV passphrase are bootstrap + transport to the registry. +- Provider credentials are written with `register-credential` and retrieved by + credential name. +- Cross-process bootstrap requires the Postgres backend. The default in-memory + backend dies with the registering CLI process and is usable only when + registration and provider calls share one process. +- With no Postgres DSN, `InMemoryConfigStore` is the intentional standalone + default. +- On active PR #96, an explicitly configured Postgres backend is authoritative + and raises `ConfigBackendUnavailableError` on import, construction, or seed + failure; this is not protected-main behavior until merge. +- Live autonomous-development/model tests use `NVIDIA_NIM_API_KEY` only in the + bounded job that calls the model. +- `COPILOT_GITHUB_TOKEN` is never a development-model credential. +- Automated review identities and credentials are separate from product + execution and must not be repurposed. + +## Evidence taxonomy and merge requirements + +Evidence is bound to the commit actually checked out. The following are not +exact-head success: queued, pending, skipped-required, cancelled, absent, +failed, predecessor-head, stale-base, author-only, status-only, synthetic-merge, +rate-limited, or infrastructure-only results. + +A protected merge requires repository policy, required checks, security gates, +zero valid unresolved findings, and a qualifying independent non-author +approval on the same unchanged head. Automated findings are triage inputs, not +human approval unless repository rules explicitly and legitimately count the +reviewing identity. + +## Deployment requirements + +### Standalone + +- mock or provider agent JSON; +- optional in-memory-only state; +- optional SQLite state and agent-pool files; +- optional in-memory credential backend for development only; +- explicit bearer tokens and loopback bind by default. + +### Modular CWL integration + +- host owns ingress, identity, tenant authorization, business persistence, + deployment, and end-user privacy obligations; +- Contextual Orchestrator owns orchestration policy and provider execution + inside its interface; +- `pg-llm-batch` owns its batch persistence/execution contract; +- naruon, inkspan, and other consumers retain their transports and schemas; +- Clearfolio remains an optional viewer integration. + +## Failure and recovery requirements + +| Failure | Required behavior | Recovery evidence | +|---|---|---| +| Invalid caller request | Fail before provider egress with stable 4xx semantics. | Contract test and no provider call. | +| Missing credential | Fail closed as not configured. | KV test and no ambient fallback. | +| Transient provider failure | Bounded retry, eligible failover, circuit update. | Trace identifies attempts and serving agent. | +| Permanent provider/caller failure | No retry storm. | Stable classified error. | +| Optional ledger export failure | Completion may continue, but export health records the loss. | Prompt-safe telemetry and flush health. | +| SQLite corruption/unavailability | Startup or write fails visibly; no fabricated persisted evidence. | Operator restores a known backup or starts an explicitly new store. | +| External batch outage | Job remains classifiable and does not become a completed result. | Poll/retry under backend contract. | +| Central review outage | Merge waits; local product and documentation work continues. | Fresh exact-head review later. | + +## Technical gaps + +- `OPENAPI_SPEC`, runtime dispatch, scopes, and endpoint prose are not generated + from one registry and have already drifted. +- Passthrough and route-streaming bypass coordinator usage accounting; route + streaming also bypasses durable state. Failed mid-stream runs are not retained. +- Workflow spend/budget and the cost ledger are unsynchronized. The ledger + treats missing price as zero, its SQL price table is dormant, and budget + admission is pre-run and non-atomic across request threads. +- Coordinator batch state is process-local; chat-batch retrieval is not + idempotent and can record duplicate usage. +- `route_p95_seconds` is exposed but is not used for dispatch. Agent selection + is deterministic tag/domain/priority scoring, not learned, price-aware, or + load-balanced. +- Protected `main` can still downgrade a configured Postgres configuration path + to memory; the active PR #96 stack instead fails closed with + `ConfigBackendUnavailableError` and does not make that behavior shipped. +- Token counting may deliberately degrade to the documented heuristic and must + qualify the resulting counts as estimated evidence. +- Retention pruning, encryption, tenancy, backup, and schema migrations for the + generic SQLite `records` store are not production complete. +- Protected main validates global provider addresses but does not yet contain + the entire DNS-pinned/strict-response boundary from PR #96. +- Production readiness and buyer-evidence endpoints are local evidence views, + not substitutes for deployed SLOs or external attestations. +- Learned routing remains planned. Adaptive reasoning, free-first fallback, and + NIM benchmark requirements are planned; PR #99, PR #94, and PR #90 are + `superseded` closed-unmerged evidence. PR #115 is an open `superseded` + scaffold rather than accepted benchmark authority. +- Active PR #111 is a partial price/admin/session slice; active PR #112 is a + release-evidence prototype rather than a trusted authority binder; active PR + #114 is a partial immediate-race experiment; and active PR #121 is a partial + liveness/readiness, inbound-framing, and trace-authority slice. Their green + branch checks do not complete their issue contracts or make them protected + product behavior. diff --git a/docs/UML.md b/docs/UML.md new file mode 100644 index 000000000..2e2df2db8 --- /dev/null +++ b/docs/UML.md @@ -0,0 +1,296 @@ +# Runtime and deployment UML + +**Document state:** `accepted_architecture` + +These diagrams are architecture-as-code. Labels use protected-main class and +resource names. Active-pull-request behavior is called out rather than drawn as +shipped runtime behavior. + +## Component topology + +```mermaid +flowchart TB + consumer["API consumer"] --> server["server.py delivery"] + operator["Operator"] --> admin["admin.py and evidence API"] + server --> coordinator["CostRoutingCoordinator"] + coordinator --> orchestrator["TaskOrchestrator"] + coordinator --> ledger["CostLedger"] + coordinator --> batch["Local or pg-llm-batch backend"] + server -. passthrough / route stream .-> orchestrator + admin --> orchestrator + orchestrator --> provider["ModelClient"] + orchestrator --> state["Optional state adapters"] + provider --> model["OpenAI-compatible provider"] +``` + +The server and admin are delivery adapters. They cannot grant model-provider +authority, change evidence status, or take ownership of host tenancy. + +## Route sequence + +```mermaid +sequenceDiagram + actor Caller + participant Server + participant Coordinator as CostRoutingCoordinator + participant Orchestrator as TaskOrchestrator + participant Provider as Model provider + + Caller->>Server: POST chat completion + Server->>Server: Authenticate and validate bounds + Server->>Coordinator: complete(messages, mode, hints) + Coordinator->>Coordinator: Choose sync and estimate tokens + Coordinator->>Orchestrator: run(messages, mode) + Orchestrator->>Orchestrator: Budget, agent, and KV credential + Orchestrator->>Provider: Bounded compatible request + alt transient provider failure + Provider-->>Orchestrator: timeout, 429, or 5xx + Orchestrator->>Orchestrator: Retry, fail over, update circuit + else permanent failure + Provider-->>Orchestrator: stable permanent error + end + Provider-->>Orchestrator: answer and optional usage + Orchestrator-->>Coordinator: route result and workflow evidence + Coordinator->>Coordinator: Append qualified usage record + Coordinator-->>Server: compatible result and cost metadata + Server-->>Caller: compatible response or validated SSE +``` + +Mock agents return before credential and network operations. The stronger +DNS-pinned and strict response parser path is `active_pr` in #96. Raw +passthrough and route streaming do not follow this coordinator sequence on +protected main; they bypass ledger recording, and streaming also bypasses +durable workflow state. + +## Conduct sequence and access lists + +```mermaid +sequenceDiagram + participant Orchestrator as TaskOrchestrator + participant Thinker as Thinker agent + participant Worker as Worker agent + participant Verifier as Verifier agent + participant Synthesizer as Synthesizer agent + + Orchestrator->>Orchestrator: Admitted complex request + Orchestrator->>Orchestrator: Template or validated generated plan + Orchestrator->>Thinker: Step 0 subtask + Thinker-->>Orchestrator: Step 0 output + Orchestrator->>Worker: Step 1 plus access [0] + Worker-->>Orchestrator: Step 1 output + Orchestrator->>Verifier: Step 2 plus access [0, 1] + Verifier-->>Orchestrator: Verdict and evidence + Orchestrator->>Synthesizer: Step 3 plus allowed outputs + Synthesizer-->>Orchestrator: Final answer + Orchestrator->>Orchestrator: Store answer and authorized trace projection +``` + +Role-specific reasoning effort and recursive depth controls are +`active_pr`/`planned`; they are not shown as protected-main authority. + +## Credential bootstrap and use + +```mermaid +sequenceDiagram + actor Deployer + participant CLI as register-credential CLI + participant KV as Credential backend + participant Runtime + participant Provider + + Deployer->>CLI: Secret over stdin + CLI->>KV: register_credential(name, value) + KV-->>CLI: Stored or explicit failure + Runtime->>KV: get_credential(name) + KV-->>Runtime: Current value or missing + alt credential present + Runtime->>Provider: Authorization at request boundary + Provider-->>Runtime: Response + else credential missing + Runtime-->>Runtime: Fail closed as not configured + end +``` + +Environment variables may select/connect/unlock the KV at bootstrap. They are +not the request-time provider credential source. The cross-process sequence +requires the durable Postgres credential backend. With the default in-memory +backend, the CLI process exits with its registry; registration must occur in the +same long-lived process as provider use. + +## Provider failover and circuit breaker + +```mermaid +sequenceDiagram + participant Orchestrator as TaskOrchestrator + participant Client as ModelClient + participant Primary as Primary provider + participant Circuit as Circuit state + participant Fallback as Eligible fallback + Orchestrator->>Orchestrator: Validate caller request and bounds + alt caller validation error + Orchestrator-->>Orchestrator: Terminate without provider dispatch + else admitted request + Orchestrator->>Circuit: Check primary availability + Circuit-->>Orchestrator: Closed or half-open + Orchestrator->>Client: Invoke primary + Client->>Primary: Bounded request + alt transient provider failure + Primary-->>Client: timeout, 429, or 5xx + Client-->>Orchestrator: Retry budget exhausted + Orchestrator->>Circuit: Record failure/open threshold + Orchestrator->>Fallback: Invoke eligible candidate + Fallback-->>Orchestrator: Valid response or classified failure + else permanent provider or configuration error + Primary-->>Client: Stable provider 4xx or configuration failure + Client-->>Orchestrator: Classified failure without client retry + Orchestrator->>Fallback: Invoke eligible candidate without client retry + else success + Primary-->>Client: Valid response + Client-->>Orchestrator: Answer and optional usage + Orchestrator->>Circuit: Reset failure state + end + end +``` + +DNS pinning, ambient-proxy and redirect rejection, and strict response parsing +are `active_pr` in #96. This diagram therefore describes the accepted control +flow while `ARCHITECTURE.md` and `TRACEABILITY.md` retain the shipped boundary. + +## Sync-versus-batch sequence + +```mermaid +sequenceDiagram + actor Caller + participant Server + participant Router as CostRoutingCoordinator + participant Orchestrator as TaskOrchestrator + participant Backend as BatchBackend + + Caller->>Server: Request plus routing hints + Server->>Router: complete(...) + Router->>Router: Policy and token estimate + alt interactive or sync decision + Router->>Orchestrator: run route or conduct + Orchestrator-->>Router: Answer and workflow identity + Router->>Router: Record qualified ledger usage + Router-->>Server: Completion + Server-->>Caller: Completion + else latency-tolerant batch decision + Router->>Backend: Submit bounded batch + Backend-->>Router: Batch job identity + Router-->>Server: Submitted state + Server-->>Caller: Submitted state + Caller->>Server: Poll or retrieve + Server->>Router: poll_batch(...) or retrieve_batch(...) + Router->>Backend: poll(...) or retrieve(...) + Backend-->>Router: State or results + opt Retrieved completion results + Router->>Router: Record qualified result usage + end + Router-->>Server: State or qualified results + Server-->>Caller: State or qualified results + end +``` + +The local backend preserves standalone operation. External job persistence and +execution are owned by the injected `pg-llm-batch` contract, but coordinator +handles and result mappings are process-local. Restart loses lookup authority; +chat result replay is not idempotent on protected main. + +## Request and provider state machine + +```mermaid +stateDiagram-v2 + [*] --> Received + Received --> Rejected: auth or validation failure + Received --> Admitted: bounds and budget pass + Admitted --> Routed: route mode + Admitted --> Planned: conduct mode + Planned --> Executing: valid bounded workflow + Routed --> Executing + Executing --> Retrying: transient failure + Retrying --> Executing: retry or failover available + Retrying --> Failed: budget or candidates exhausted + Executing --> Failed: permanent failure + Executing --> Completed: valid answer + Completed --> Persisted: optional store succeeds + Completed --> DegradedEvidence: optional export fails + Persisted --> [*] + DegradedEvidence --> [*] + Rejected --> [*] + Failed --> [*] +``` + +`DegradedEvidence` is never converted into complete durable evidence. A model +answer and its export health remain separate facts. + +## Evidence and merge authority + +```mermaid +stateDiagram-v2 + [*] --> CandidateHead + CandidateHead --> DeterministicEvidence: tests and security execute + DeterministicEvidence --> ReviewEvidence: exact-head reviews execute + ReviewEvidence --> Approved: eligible independent approval + Approved --> Mergeable: all protected gates agree + Mergeable --> ProtectedMain: protected merge + CandidateHead --> Blocked: failed, absent, stale, or synthetic evidence + DeterministicEvidence --> Blocked: valid finding or missing gate + ReviewEvidence --> Blocked: unresolved finding + Blocked --> CandidateHead: deliberate new head +``` + +A status, check, model review, and human approval are distinct. No transition +may manufacture another authority. + +## Deployment topology + +```mermaid +flowchart TB + subgraph Host["Host-owned boundary"] + ingress["Ingress and identity"] + tenancy["Tenant and purpose policy"] + business["Business records"] + end + subgraph Orchestrator["Contextual Orchestrator"] + api["Compatible API"] + policy["Orchestration policy"] + provider_adapter["Provider adapter"] + evidence["Trace, cost, and audit evidence"] + end + subgraph Dependencies["Optional dependencies"] + kv["Credential registry"] + provider["OpenAI-compatible provider"] + batch["pg-llm-batch"] + viewer["Clearfolio"] + end + ingress --> api + tenancy --> api + api --> policy + policy --> evidence + policy --> provider_adapter + provider_adapter --> kv + provider_adapter --> provider + policy --> batch + evidence --> viewer + business -. purpose-bound request .-> ingress +``` + +The dotted edge is a request projection, not a transfer of business-record +ownership. + +## Degraded-mode topology + +```mermaid +flowchart LR + request["Admitted request"] --> provider["Provider execution"] + provider --> answer["Qualified answer"] + answer --> optional["Optional state, ledger, or evidence export"] + optional -->|success| durable["Durable qualified evidence"] + optional -->|failure| degraded["Degraded evidence state"] + degraded --> operator["Operator alert and incident runbook"] + degraded -. never promoted .-> blocked["Release evidence incomplete"] +``` + +Optional persistence failure may leave a model answer usable only when the +contract permits it, but it cannot be reported as complete durable evidence. diff --git a/docs/adr/0001-route-conduct-test-time-compute.md b/docs/adr/0001-route-conduct-test-time-compute.md new file mode 100644 index 000000000..f038b9918 --- /dev/null +++ b/docs/adr/0001-route-conduct-test-time-compute.md @@ -0,0 +1,74 @@ +# ADR-0001: Route and conduct test-time compute + +## Status + +`implemented_on_protected_main` + +## Context and decision drivers + +One compatible endpoint must handle simple requests economically and complex +requests with explicit decomposition and verification. Fugu, Conductor, and +TRINITY show complementary routing and orchestration patterns, but they do not +justify spending more calls on every request or claiming learned behavior. +Correctness, evidence, controllability, reliability, and comparable budgets are +primary drivers; latency is a measured guardrail. + +## Considered alternatives + +- always call one model: simple and cheap, but cannot expose structured + verification or multi-agent evidence; +- always run a fixed multi-agent workflow: auditable, but wastes compute and + confounds quality comparisons; +- learned coordinator immediately: unsupported without a versioned evaluation + set and operational reward data; +- deterministic route/conduct split with measurable policy: selected. + +## Decision + +`TaskOrchestrator.complete()` chooses `route` or `conduct` from explicit caller +mode and a snapshotted policy. Route selects one eligible worker. Conduct +executes a bounded template or validated generated plan with explicit roles and +access lists. New recursion, topology, or reasoning-effort knobs require hard +call/token caps and comparable-budget ablations. + +## Consequences + +The standalone runtime stays deterministic and testable. Deep orchestration may +improve difficult tasks but has higher cost and cannot claim live synthesizer +streaming before synthesis completes. Policy changes require evaluation replay. + +## Failure and recovery + +Invalid generated plans fall back to the bounded template. Budget exhaustion +fails before extra provider calls. If conduct quality is not better under a +comparable budget, revert affected workload cells to route or the last accepted +policy. + +## Security, privacy, and governance impact + +More steps create more provider exposure. Access lists, call bounds, provider +exclusions, and purpose-bound payload minimization apply to every step. A deeper +workflow never expands tool or credential authority. + +## Compatibility and migration + +The public API remains one model-like surface. New policy fields default to +current deterministic behavior and require trace versioning. + +## Verification and acceptance + +Route/conduct contract tests, access-list tests, fixed-task comparable-budget +evaluation, per-cell call/token evidence, uncertainty, and exact-head coverage +are required. Learned replacement additionally needs repeatable superiority over +the deterministic baseline. + +## Rollback and supersession + +Rollback selects the prior policy without changing request schema. Supersede +only with an ADR documenting evaluation data, budget parity, failure behavior, +and migration. + +## References + +Fugu Team, Sakana AI (2026); Nielsen et al. (2025); Xu et al. (2025). Full APA +7 entries are in [the reference index](../REFERENCES.md). diff --git a/docs/adr/0002-provider-neutral-transport-trust.md b/docs/adr/0002-provider-neutral-transport-trust.md new file mode 100644 index 000000000..e70414d95 --- /dev/null +++ b/docs/adr/0002-provider-neutral-transport-trust.md @@ -0,0 +1,74 @@ +# ADR-0002: Provider-neutral interface and transport trust + +## Status + +`accepted_architecture` — the provider-neutral HTTPS/global-address boundary is +`implemented_on_protected_main`; DNS pinning, proxy/redirect rejection, and the +strict response boundary are `active_pr` in #96. + +## Context and decision drivers + +The gateway must swap OpenAI-compatible providers without provider SDK lock-in +while preventing stored SSRF, credential forwarding, ambiguous responses, and +unbounded resource use. Validation that is disconnected from socket selection +does not stop DNS rebinding. + +## Considered alternatives + +- provider SDK per vendor: richer features, but fragmented authority and + dependency surface; +- validate URL once, then use a default opener: vulnerable to resolver and + redirect/proxy changes; +- trusted internal proxy only: useful deployment option but not a standalone + security guarantee; +- compatible HTTP contract with end-to-end pinned trust: selected architecture. + +## Decision + +Model configuration contains a compatible base URL and credential name. +Production egress is HTTPS, globally routable, optionally allowlisted, bounded, +and credentialed only at the final request boundary. The accepted transport +retains validation-time addresses through connection establishment, preserves +host/TLS authority, disables ambient proxies and redirects, bounds cumulative +response bytes, and strictly validates JSON/JSONL/SSE framing. Until #96 merges, +only protected-main controls may be claimed shipped. + +## Consequences + +Providers remain swappable. Some enterprise proxies require an explicit, +reviewed adapter rather than ambient behavior. Strict parsing may reject +provider extensions outside the documented subset. + +## Failure and recovery + +Private/non-global destinations, missing credentials, TLS failure, redirects, +ambiguous framing, oversized bodies, malformed JSON/SSE, and non-finite values +fail closed. Transient network/provider errors alone enter bounded retry/failover. + +## Security, privacy, and governance impact + +The design limits credential exfiltration and internal-network reach. Provider +content remains untrusted and cannot grant tool, review, or host authority. + +## Compatibility and migration + +Mock providers remain networkless. Compatible providers that rely on redirects, +proxies, nonstandard JSON, or private destinations need an explicit deployment +contract rather than silent compatibility. + +## Verification and acceptance + +Tests cover global-address policy, DNS rebinding, IPv4/IPv6, TLS SNI and +certificate identity, redirect/proxy leakage, credential revocation, retries, +cleanup, body/framing bounds, strict JSON/JSONL/SSE, and redacted errors. + +## Rollback and supersession + +Do not roll back by weakening validation. Disable a faulty provider or revert to +the last accepted pinned implementation. A replacement requires equivalent +security tests and an explicit authority map. + +## References + +Bray (2017), Fielding et al. (2022), Rescorla (2018), OWASP Foundation (n.d.). +See [the reference index](../REFERENCES.md). diff --git a/docs/adr/0003-workflow-access-and-reasoning-control.md b/docs/adr/0003-workflow-access-and-reasoning-control.md new file mode 100644 index 000000000..cd54f9ecf --- /dev/null +++ b/docs/adr/0003-workflow-access-and-reasoning-control.md @@ -0,0 +1,69 @@ +# ADR-0003: Explicit workflow access and role reasoning control + +## Status + +`accepted_architecture` — explicit workflow steps/access lists are +`implemented_on_protected_main`; adaptive role-specific reasoning is +`active_pr` in #99. + +## Context and decision drivers + +Multi-agent quality depends on topology, task decomposition, worker assignment, +and information flow. Giving every worker the whole transcript increases cost, +PII exposure, prompt-injection reach, and correlated error. Conductor and +TRINITY provide useful explicit workflow/role abstractions. + +## Considered alternatives + +- shared full transcript: easiest, but violates least context; +- fixed four steps only: deterministic, but not task-adaptive; +- unrestricted generated workflows: flexible, but unsafe and unbounded; +- bounded validated plans with explicit access and optional reviewed effort + profiles: selected. + +## Decision + +Every workflow step declares role, agent, natural-language subtask, and prior +step IDs it may access. Generated plans are structurally validated and bounded; +invalid plans use the template. Reasoning effort, recursion, and decomposition +are policy values, not provider-output authority, and must preserve common +budgets in evaluations. + +## Consequences + +Information flow is inspectable and testable. Some useful context must be +deliberately listed. Provider-specific effort mappings remain adapters behind a +provider-neutral profile. + +## Failure and recovery + +Unknown agents, forward references, cycles, invalid roles, excessive depth, or +budget overflow reject/fallback before execution. A faulty effort adapter falls +back to the last accepted profile, not an unbounded provider default. + +## Security, privacy, and governance impact + +Access lists reduce unnecessary PII and hostile-output propagation. They do not +sanitize visible content or grant tools; integrating hosts still enforce purpose +and tool authority. + +## Compatibility and migration + +Existing template workflows remain the default. New profile fields are optional +and trace-versioned. #99 evidence does not transfer before its stack merges. + +## Verification and acceptance + +Access-list visibility tests, plan parser/property tests, cycle/forward-reference +rejection, per-role payload tests, comparable-budget ablations, and exact-head +coverage are required. + +## Rollback and supersession + +Disable generated/adaptive policy and return to the bounded template. Supersede +only with a flow-control model that preserves explicit inspectable authority. + +## References + +Nielsen et al. (2025); Xu et al. (2025); Fugu Team, Sakana AI (2026). See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0004-kv-credential-bootstrap.md b/docs/adr/0004-kv-credential-bootstrap.md new file mode 100644 index 000000000..8a7c10570 --- /dev/null +++ b/docs/adr/0004-kv-credential-bootstrap.md @@ -0,0 +1,67 @@ +# ADR-0004: KV credential registry and bootstrap-only environment + +## Status + +`implemented_on_protected_main` + +## Context and decision drivers + +Ambient request-time environment lookup makes credential source and rotation +unclear and spreads secret authority across the process. The product needs one +auditable retrieval seam that works offline and with encrypted production +storage. + +## Considered alternatives + +- read provider keys with `os.getenv` on every request: rejected; +- require one external secret product: conflicts with standalone operation; +- put secrets in agent JSON: rejected because configuration becomes secret data; +- pluggable KV registry with explicit bootstrap: selected. + +## Decision + +`ModelAgent` stores a credential name. `get_credential` resolves its value from +an in-memory development backend or pgcrypto-encrypted Postgres backend. +Environment variables may select/connect/unlock the KV and may feed a one-shot +bootstrap CLI, but runtime provider execution does not use ambient environment +fallback. + +## Consequences + +Mock/offline tests need no secret. Production deployment must operate and back +up the credential registry and protect its passphrase. The legacy +`api_key_env` field is only a credential-name alias. + +## Failure and recovery + +Missing/unavailable credentials fail closed before provider egress. Rotate by +revoking at the provider, updating KV, and verifying stale processes cannot use +the old value. + +## Security, privacy, and governance impact + +Secret values stay out of agent files, prompts, traces, logs, and telemetry. +Least-privilege database roles, encryption keys, rotation, and audit are +deployment obligations. + +## Compatibility and migration + +Existing agent JSON remains readable. Migrate secret values into KV before +removing old environment injection. Do not expose the old value through a +compatibility log or response. + +## Verification and acceptance + +Tests cover mock bypass, missing credentials, legacy-name behavior, backend +selection, stdin bootstrap, no ambient fallback, encryption SQL, and redacted +errors. + +## Rollback and supersession + +Rollback selects a previous KV backend, never raw request-time environment +lookup. A dedicated secret manager may supersede Postgres behind the same seam. + +## References + +NIST SP 800-218 and ISO/IEC 27001:2022; see +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0005-sync-batch-pg-llm-batch.md b/docs/adr/0005-sync-batch-pg-llm-batch.md new file mode 100644 index 000000000..a3c1143bb --- /dev/null +++ b/docs/adr/0005-sync-batch-pg-llm-batch.md @@ -0,0 +1,76 @@ +# ADR-0005: Standalone sync/batch routing with optional pg-llm-batch + +## Status + +`implemented_on_protected_main` + +## Context and decision drivers + +Interactive requests and bulk evaluation/embedding workloads have different +latency, throughput, and price characteristics. The orchestrator must keep one +cost/policy boundary without making standalone use depend on an external batch +service. + +## Considered alternatives + +- sync only: simple but uneconomical for latency-tolerant work; +- make `pg-llm-batch` mandatory: breaks standalone modularity; +- duplicate cost policy in both services: creates contradictory evidence; +- shared routing contract with local and injected backends: selected. + +## Decision + +`RoutingPolicy` uses explicit hints and KV thresholds. Interactive work stays on +sync execution. Latency-tolerant work uses a `BatchBackend` or embedding backend. +Local in-process implementations preserve offline/standalone behavior; +`PgLlmBatchBackend` and configuration adapters preserve external ownership. +Usage from ordinary coordinator sync completion and coordinator-completed +batch retrieval enters the same prompt-safe ledger contract. The accepted +target requires sync completion, batch retrieval, passthrough, and route +streaming to use one qualified usage-evidence contract. Missing provider usage +is `unknown`; a path that bypasses the evidence writer is `not_recorded`. +Either status is excluded from cost comparison until reconciled. On protected +main, passthrough and route streaming bypass the ledger, and submit/poll alone +does not record completion usage; the unified target is not presented as +shipped. + +## Consequences + +Callers receive observable job states. External adapters add operational +dependencies but do not remove the interactive path. Cost comparison can use +one attribution vocabulary only for the coordinator paths that record usage. +Coordinator job handles and replay guards remain process-local. + +## Failure and recovery + +Submit/poll/retrieve failures remain non-success job states. Oversized embedding +inputs split under explicit limits. External outage does not convert a job to +complete or prevent local interactive requests. + +## Security, privacy, and governance impact + +The external backend receives only its versioned payload and purpose metadata. +It owns transport, authorization, persistence, retention, and job tenancy under +its contract; the orchestrator cannot infer those controls. + +## Compatibility and migration + +The local backend is the default. Adopting `pg-llm-batch` injects adapters and +config rather than changing caller schema. Backend identifiers and result +semantics remain stable. + +## Verification and acceptance + +Tests cover decision reasons, sync and batch use, submit/poll/retrieve, +embeddings splitting/reduction, attribution, malformed/partial results, and +dependency failure. + +## Rollback and supersession + +Route affected work to the local backend or disable latency-tolerant submission. +Supersede only with an adapter preserving job/evidence and standalone contracts. + +## References + +Chen et al. (2023); Ding et al. (2024); Ong et al. (2024). See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0006-honest-cost-and-benchmark-evidence.md b/docs/adr/0006-honest-cost-and-benchmark-evidence.md new file mode 100644 index 000000000..946083e91 --- /dev/null +++ b/docs/adr/0006-honest-cost-and-benchmark-evidence.md @@ -0,0 +1,84 @@ +# ADR-0006: Honest cost and benchmark evidence + +## Status + +`accepted_architecture` + +## Context and decision drivers + +Routing and orchestration decisions depend on cost, quality, latency, and token +evidence. Provider-reported usage, local token estimates, configured prices, +benchmark scores, and external invoices have different authorities. Combining +them without provenance can make an unpriced or weakly measured path appear +better than it is. + +## Considered alternatives + +- report one blended cost number: simple, but conceals unknowns and estimates; +- trust provider marketing or a single benchmark run: current, but neither + reproducible nor workload-specific; +- block all routing when any field is unknown: safe but unnecessarily removes + useful partial evidence; +- preserve source, method, uncertainty, and missingness per value: selected. + +## Decision + +Every cost, token, latency, quality, and benchmark fact identifies its source as +provider-reported, locally measured, configured, estimated, unknown, or +external. Unknown price remains unknown rather than zero. Comparisons use fixed, +versioned tasks and scorers, repeated cells, comparable call/token budgets, and +full model/provider/policy assignments. Repository results never claim a live +price, contractual bill, certification, or general superiority. + +The accepted evidence vocabulary carries the same seven attribution dimensions +through every writer and export path: `account`, `service`, `upstream_api`, +`model_name`, `team`, `group`, and `company`. Mode-by-mode completeness +tests cover sync completion, batch retrieval, passthrough, and route streaming; +a mode that cannot supply a fact records qualified missingness instead of +silently omitting the dimension. + +Protected main does not yet satisfy the whole decision. Workflow-derived +spend/budget and `CostLedger` are separate authorities; the active `PriceBook` +uses ConfigStore while SQL `llm_price_entries` is dormant; missing ledger price +becomes `0.0`; and `cheapest_upstream()` is not used for selection. These facts +are recorded as gaps, not described as free or cost-optimized routing. + +## Consequences + +Evidence is more verbose and some comparisons remain inconclusive. Operators +can nevertheless distinguish accounting facts from routing estimates and can +replay the exact policy decision. + +## Failure and recovery + +Missing usage or prices produce explicitly incomplete evidence. An export +failure does not fabricate persistence. Recovery reconciles by immutable run +identity and marks irrecoverable gaps; it never imputes them as measurements. + +## Security, privacy, and governance impact + +Usage records exclude prompts, answers, secrets, and unnecessary PII. Benchmark +artifacts contain only the minimum reproducibility data and cannot confer +review, release, or buyer-acceptance authority. + +## Compatibility and migration + +Existing numeric fields remain readable, but new writers supply provenance and +measurement status. Readers treat absent legacy provenance as unknown. + +## Verification and acceptance + +Tests cover reported versus estimated tokens, configured versus unknown price, +unpriced model lists, prompt-safe export, degraded-store telemetry, repeated +benchmark cells, assignment completeness, and comparable budgets. + +## Rollback and supersession + +Rollback may disable a faulty estimator or benchmark policy but must retain raw +qualified evidence. Supersession requires a documented measurement model and a +reproducible migration of historical classifications. + +## References + +Chen et al. (2023), Ding et al. (2024), and Ong et al. (2024). Full APA 7 +entries are in [the reference index](../REFERENCES.md). diff --git a/docs/adr/0007-free-first-fallback.md b/docs/adr/0007-free-first-fallback.md new file mode 100644 index 000000000..3dc355c4c --- /dev/null +++ b/docs/adr/0007-free-first-fallback.md @@ -0,0 +1,69 @@ +# ADR-0007: Free-first fallback without invented availability + +## Status + +`active_pr` — implementation and evidence are isolated to PR #94 and are not +protected-main behavior. + +## Context and decision drivers + +Some providers expose zero-price or promotional model access, but price, +capacity, eligibility, and policy change independently. A free-first policy can +reduce spend only when the candidate is genuinely eligible and the fallback +does not weaken quality, privacy, reliability, or budget controls. + +## Considered alternatives + +- always select the cheapest configured price: ignores availability and quality; +- hard-code a provider's free model list: rapidly stale and provider-specific; +- treat unknown price as free: financially misleading; +- select only reviewed eligible zero-price candidates, then use the normal + bounded fallback policy: selected. + +## Decision + +Free-first is an opt-in policy over operator-supplied price and eligibility +evidence. A candidate must be enabled, non-excluded, compatible, and explicitly +classified as zero-price for the relevant dimensions. Unknown price or +availability is not free. Failures use the existing bounded retry/failover and +budget path; free status never grants extra context or authority. + +## Consequences + +Some nominally free opportunities will be skipped when evidence is incomplete. +Selection remains portable and auditable rather than depending on a vendor +catalog embedded in domain code. + +## Failure and recovery + +Stale price, missing eligibility, quota failure, or provider degradation removes +the candidate from that decision and records the reason. Recovery refreshes +operator evidence and replays fixed evaluation cells before re-enabling policy. + +## Security, privacy, and governance impact + +A lower price cannot override provider allowlists, data-use restrictions, +credential policy, tenant purpose, or model exclusions. Price evidence carries +source and review time without including secrets. + +## Compatibility and migration + +The default policy remains protected-main selection. Enabling free-first adds a +versioned policy field; clients that do not supply it retain current behavior. + +## Verification and acceptance + +Acceptance requires exact-head tests for zero/unknown/non-zero price, stale +availability, exclusions, quota failure, fallback bounds, stable traces, +prompt-safe evidence, and comparable-budget quality. PR #94 must independently +satisfy protected repository gates before this status changes. + +## Rollback and supersession + +Disable the policy flag and retain its decision trace. A replacement must state +price authority, freshness, eligibility, quality floor, and failure semantics. + +## References + +Chen et al. (2023), Ding et al. (2024), and Ong et al. (2024). See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0008-state-persistence-and-retention.md b/docs/adr/0008-state-persistence-and-retention.md new file mode 100644 index 000000000..c8876f757 --- /dev/null +++ b/docs/adr/0008-state-persistence-and-retention.md @@ -0,0 +1,73 @@ +# ADR-0008: Explicit state persistence and retention authority + +## Status + +`accepted_architecture` — in-memory defaults and opt-in SQLite persistence are +`implemented_on_protected_main`; production retention, tenancy, encryption, and +migrations remain incomplete. + +## Context and decision drivers + +Standalone operation benefits from zero-infrastructure defaults, while audits +and restart recovery may require durable state. Workflow payloads can contain +prompts, answers, and PII, so silently enabling persistence would create +security, privacy, recovery, and records-management obligations. + +## Considered alternatives + +- always persist to a database: durable but violates the standalone and + data-minimization defaults; +- memory only: simple but cannot support requested restart evidence; +- persist only telemetry: insufficient for authorized workflow recovery; +- default to memory and enable each store explicitly with operator authority: + selected. + +## Decision + +Process memory is the default. SQLite state and agent overlays, the SQL cost +ledger, and the Postgres credential backend are separate opt-in adapters with +separate ownership. Enabling payload persistence requires an operator-defined +purpose, audience, encryption, retention, deletion, backup, residency, and +recovery policy. Generic runtime JSON is not presented as the normalized target. + +## Consequences + +A default restart loses ephemeral state by design. Durable deployments must do +more operational work, but can choose the minimum store needed and can reason +about its data classification. + +## Failure and recovery + +Store failure is visible and cannot be converted into a claim of durable +evidence. Operators preserve a failed file read-only, restore a verified backup +or initialize an explicitly new store, reconcile gaps, and run restart tests. + +## Security, privacy, and governance impact + +Generic `records.payload` may contain sensitive content and currently lacks +field encryption, automatic pruning, tenant partitioning, and subject-rights +workflows. Those gaps block production durability claims unless the host +supplies compensating controls. + +## Compatibility and migration + +Existing in-memory and SQLite deployments remain supported. Any move to the +normalized target uses expand, backfill, verify, switch, and contract phases +with reader compatibility and rollback. + +## Verification and acceptance + +Tests cover disabled-by-default behavior, restart recovery, schema identity, +parameter binding, bounds, corruption/unavailability, retention hooks, backup +restore, and reconciliation between generic and normalized representations. + +## Rollback and supersession + +Disable the adapter only after preserving or deliberately disposing of data +under policy. Supersession requires a data migration, dual-read/write boundary, +reconciliation proof, and tested rollback. + +## References + +NIST (2022, 2024b) and ISO/IEC 27001:2022. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0009-purpose-bound-pii-protection.md b/docs/adr/0009-purpose-bound-pii-protection.md new file mode 100644 index 000000000..5f25ec045 --- /dev/null +++ b/docs/adr/0009-purpose-bound-pii-protection.md @@ -0,0 +1,73 @@ +# ADR-0009: Purpose-bound PII protection + +## Status + +`accepted_architecture` + +## Context and decision drivers + +Authorized business tasks may require names, account details, or other personal +data. Blanket masking can destroy the task's meaning, while copying full payloads +into traces, analytics, caches, or broad operator views creates avoidable risk. +Protection must follow purpose, audience, authority, and lifecycle. + +## Considered alternatives + +- mask every recognized identifier before orchestration: reduces utility and + can corrupt required facts; +- retain complete payloads everywhere for debugging: operationally convenient + but violates minimization; +- rely only on provider policy: delegates responsibilities the host still owns; +- preserve minimum authorized payload on the execution path and minimize every + derived projection: selected. + +## Decision + +The integrating host establishes purpose, legal basis, tenant/user authority, +provider eligibility, and subject-rights handling. The orchestrator passes only +the minimum authorized content to selected providers and roles. Telemetry and +cost records exclude raw prompts/answers; traces, persistence, caches, previews, +and operator views have distinct audience controls, retention, and redaction. +Masking is a projection control, not authorization or encryption. + +## Consequences + +Deployments need data classification and audience policy rather than one global +redaction switch. Correctly authorized workflows preserve business meaning, +while broad evidence surfaces contain less sensitive data. + +## Failure and recovery + +Unknown purpose, authority, provider eligibility, or trace audience fails closed +for the affected exposure. An exposure incident triggers provider containment, +credential review, deletion/retention procedures, evidence preservation, impact +assessment, and host-owned notification obligations. + +## Security, privacy, and governance impact + +This decision applies minimization, least authority, separation of telemetry +from content, and lifecycle controls. It does not claim that the repository +alone satisfies a jurisdiction, DPA, or certification. + +## Compatibility and migration + +Existing request semantics remain. Integrations add purpose/tenant authority at +their boundary and must partition or disable caches and persistence until those +keys participate in authorization. + +## Verification and acceptance + +PII-bearing fixtures verify preservation on the authorized model path, +exclusion from prompt-safe telemetry, audience-limited traces, cache isolation, +retention/deletion behavior, redacted failures, and provider restrictions. + +## Rollback and supersession + +Rollback means narrowing or disabling the affected projection, never restoring +unbounded copying. Supersession requires a privacy threat model, data-flow map, +compatibility plan, and deployment-specific legal review. + +## References + +NIST AI RMF 1.0, NIST AI 600-1, ISO/IEC 27001:2022, and ISO/IEC 42001:2023. +See [the reference index](../REFERENCES.md). diff --git a/docs/adr/0010-independent-review-and-evidence.md b/docs/adr/0010-independent-review-and-evidence.md new file mode 100644 index 000000000..6eac41e26 --- /dev/null +++ b/docs/adr/0010-independent-review-and-evidence.md @@ -0,0 +1,85 @@ +# ADR-0010: Independent review and evidence authority + +## Status + +`accepted_architecture` + +## Context and decision drivers + +Checks, commit statuses, automated reviews, human reviews, unresolved findings, +mergeability, and protected merge answer different questions. Treating any one +as a substitute for the others can ship stale, synthetic, author-only, or +unreviewed work. + +## Considered alternatives + +- merge when named workflows are green: ignores head identity and approval; +- treat an automated approval/status as independent human approval: conflates + evidence identities; +- reconstruct repository rules from prose: likely to drift from GitHub; +- bind every evidence item to exact head and retain protected merge as final + authority: selected. + +## Decision + +Each candidate records contributor head, live base, checkout identity, job +conclusion, review identity, unresolved threads, and merge authority separately. +Queued, pending, skipped-required, cancelled, absent, failed, predecessor-head, +stale-base, author-only, status-only, synthetic-merge, rate-limited, and +infrastructure-only results are not exact-head success. + +Before any merge or auto-merge mutation, GitHub's live aggregate +`reviewDecision` must be `APPROVED` for the unchanged head. A missing decision, +`REVIEW_REQUIRED`, or `CHANGES_REQUESTED` blocks the mutation even when branch +protection currently allows zero approvals. An eligible independent non-author +approval must also be present; the aggregate field is evidence of the combined +repository state, not a substitute reviewer. A completed, successful, +structured same-head Strix report is separately required. Queued, in-progress, +neutral/no-report, cancelled, skipped, absent, or predecessor-head Strix states +block merge. + +Zero valid unresolved findings and every required exact-head check are required +in addition to those review and security gates. GitHub's protected operation is +the final merge authority. + +## Consequences + +Changes may wait when the review control plane is degraded. Safe local work, +tests, documentation, and non-conflicting branches continue without weakening +the gate or fabricating evidence. + +## Failure and recovery + +Stale or ambiguous evidence blocks merge only. A new head invalidates evidence +according to repository policy and reacquires it. If the aggregate review state +regresses or a required check becomes incomplete after auto-merge is queued, +automation disables that queued mutation and starts exact-head verification +again. Review and Strix outages are retried later; workflows do not rewrite +themselves or reduce required contexts. + +## Security, privacy, and governance impact + +Reviewer credentials, development-model credentials, and product credentials +are separate. Automation cannot impersonate a human or alter branch protection +to approve its own work. + +## Compatibility and migration + +Existing CI remains evidence-producing infrastructure. Adoption adds explicit +head/base capture and removes prose that promotes statuses into approvals. + +## Verification and acceptance + +Acceptance scenarios cover exact contributor head, synthetic merge, new commits, +stale base, missing checks, unresolved threads, author review, automated review, +independent approval, and protected merge refusal. + +## Rollback and supersession + +The gate may be made stricter without migration. Any relaxation requires a +security ADR, ruleset-owner approval, and equivalent independent-control proof. + +## References + +NIST SP 800-218 and NIST SP 800-218A. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0011-release-coverage-and-provenance.md b/docs/adr/0011-release-coverage-and-provenance.md new file mode 100644 index 000000000..f45ed027e --- /dev/null +++ b/docs/adr/0011-release-coverage-and-provenance.md @@ -0,0 +1,71 @@ +# ADR-0011: Release coverage and provenance + +## Status + +`accepted_architecture` + +## Context and decision drivers + +A unit-test pass does not prove branch behavior, public documentation, package +installability, dependency integrity, artifact identity, migration recovery, or +independent acceptance. Release claims must bind these views to one unchanged +protected revision. + +## Considered alternatives + +- release from any green feature branch: fast but bypasses integrated authority; +- accept line coverage alone: misses branch behavior and excluded production; +- trust a built artifact without source/provenance linkage: irreproducible; +- require one protected revision with complete functional, security, package, + provenance, review, and recovery evidence: selected. + +## Decision + +Owned production code reaches 100% statement and branch coverage and the +repository's public-docstring target without excluding executable behavior to +improve a metric. A release candidate also passes realistic integration, fuzz, +security, compatibility, build/install/import, SBOM, provenance, +reproducibility, migration/rollback, and independent-review gates. Version, +changelog, source revision, artifacts, and published identity agree. + +## Consequences + +Release preparation is stricter than ordinary development and may reveal real +defects late in a feature branch. Coverage gaps produce tests or fixes, not +weakened thresholds. + +## Failure and recovery + +Any changed head, missing/failed gate, artifact mismatch, unresolved finding, or +rollback failure blocks publication. Repair creates a new candidate and repeats +all head-bound evidence. A bad release is stopped, identified exactly, rolled +back compatibly, and reconciled. + +## Security, privacy, and governance impact + +SBOM and provenance reduce supply-chain ambiguity. Logs and artifacts still +exclude secrets and unnecessary PII. Repository evidence does not manufacture +external certification, penetration testing, SLO, or buyer signature. + +## Compatibility and migration + +Existing development tests remain fast feedback. Release workflows add gates at +protected-main and artifact boundaries without requiring live credentials in +offline tests. + +## Verification and acceptance + +The release manifest binds commit, tag, version, changelog, package hashes, +SBOM, provenance, coverage, docstrings, test/security/fuzz results, reviewer +state, migration evidence, and reproducible install/smoke output. + +## Rollback and supersession + +Rollback restores a known compatible artifact and schema while preserving the +failed identity for incident analysis. Supersession requires equal or stronger +source-to-artifact and independent-control guarantees. + +## References + +NIST SP 800-218 and NIST SP 800-218A. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0012-standalone-and-cwl-boundary.md b/docs/adr/0012-standalone-and-cwl-boundary.md new file mode 100644 index 000000000..e38979b1b --- /dev/null +++ b/docs/adr/0012-standalone-and-cwl-boundary.md @@ -0,0 +1,70 @@ +# ADR-0012: Standalone product and explicit CWL boundary + +## Status + +`implemented_on_protected_main` + +## Context and decision drivers + +Contextual Orchestrator is both an independently useful service/library and a +module in the ContextualWisdomLab ecosystem. Hidden dependence on a central +control plane would break offline tests, local adoption, failure isolation, and +ownership clarity; duplicating host identity or business data would create a +conflicting authority. + +## Considered alternatives + +- require the full CWL stack: integrated but not independently deployable; +- copy identity, tenancy, and business records into this service: convenient + locally but creates ownership and synchronization conflicts; +- expose only a library with no service boundary: limits compatible adoption; +- keep standalone defaults and add explicit, optional host/adaptor contracts: + selected. + +## Decision + +The core runs offline with mock agents and in memory, and can run as a compatible +HTTP service with configured providers. Optional SQLite, SQL/KV, +`pg-llm-batch`, Clearfolio, naruon, inkspan, and other CWL integrations enter +through explicit interfaces. The host owns ingress identity, tenant/purpose +authorization, business records, deployment, and end-user privacy unless a +versioned contract delegates a specific responsibility. + +## Consequences + +Basic operation stays dependency-light and testable. Integrations must translate +and authorize at their boundary rather than importing implicit global state. + +## Failure and recovery + +An optional CWL dependency outage degrades only its declared capability. The +standalone route/library path remains available when safe. Recovery revalidates +the adapter contract and never backfills fabricated evidence. + +## Security, privacy, and governance impact + +Authority and data ownership remain local to the system that has purpose and +tenant context. A viewer, batch service, or orchestrator response does not grant +host access or merge/release authority. + +## Compatibility and migration + +Adapters are optional and versioned. Breaking a host boundary requires staged +dual compatibility, ownership reconciliation, and coordinated rollback. + +## Verification and acceptance + +Offline install/import/mock tests run without CWL dependencies. Contract tests +cover each adapter, missing/degraded dependency behavior, data minimization, +authorization handoff, and independent startup/recovery. + +## Rollback and supersession + +Disable the adapter and retain standalone behavior. Supersession must preserve +an independent mode or explicitly reclassify the product with migration, +availability, and ownership evidence. + +## References + +NIST SP 800-218 and ISO/IEC 42001:2023. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0013-database-naming-and-migration.md b/docs/adr/0013-database-naming-and-migration.md new file mode 100644 index 000000000..25de6d33d --- /dev/null +++ b/docs/adr/0013-database-naming-and-migration.md @@ -0,0 +1,69 @@ +# ADR-0013: Descriptive database naming and evidence-driven migration + +## Status + +`accepted_architecture` + +## Context and decision drivers + +The repository naming contract requires descriptive two-or-more-word +snake_case. Protected main also has a legacy one-word SQLite table, `records`, +while [`database_design.sql`](../database_design.sql) describes a normalized target that runtime does +not create. Renaming or applying target DDL without compatibility evidence risks +data loss and misleading architecture claims. + +## Considered alternatives + +- silently call the target SQL current: inaccurate; +- rename `records` in place: breaks existing stores and rollback; +- exempt all database objects from naming rules: removes useful consistency; +- document the exception and migrate through expand/backfill/verify/contract: + selected. + +## Decision + +New owned database identifiers use descriptive two-or-more-word snake_case, +except externally fixed standards. `records` is explicit technical debt and +remains readable until a migration introduces a descriptive replacement such as +`runtime_records`, backfills idempotently, verifies counts/content, supports a +bounded compatibility window, and contracts only after rollback is safe. Actual, +external, in-memory, active-PR, and target schemas remain separately labeled. + +## Consequences + +The legacy name persists temporarily, but no new ambiguous one-word objects are +added. Documentation reflects physical truth and migrations cost more upfront. + +## Failure and recovery + +Any schema, backfill, reconciliation, or reader-compatibility failure stops +before contract. Recovery returns reads/writes to the last compatible schema, +restores a verified backup if necessary, and records incomplete rows explicitly. + +## Security, privacy, and governance impact + +Migrations preserve least-privilege roles, encryption, retention, deletion, +tenant boundaries, and audit identity. Backfill logs never print payloads or +credentials. + +## Compatibility and migration + +Use expand, dual read/write where needed, idempotent backfill, reconciliation, +switch, observation, and contract. Each supported prior version has a tested +reader or an explicit upgrade boundary. + +## Verification and acceptance + +Tests introspect physical schemas, enforce new-name rules, migrate realistic old +stores, verify row/content parity and indexes, exercise interruption/resume, +restore backups, and prove rollback before contract. + +## Rollback and supersession + +Rollback is mandatory until contract. A later ADR may remove the compatibility +path only after retention and deployed-version evidence show it is safe. + +## References + +NIST SP 800-218 and ISO/IEC 27001:2022. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0014-scientific-computation-ownership.md b/docs/adr/0014-scientific-computation-ownership.md new file mode 100644 index 000000000..073c0e606 --- /dev/null +++ b/docs/adr/0014-scientific-computation-ownership.md @@ -0,0 +1,72 @@ +# ADR-0014: Scientific computation ownership + +## Status + +`out_of_scope` + +## Context and decision drivers + +The wider CWL portfolio may include mathematical, financial, psychometric, or +GPU-accelerated kernels. Contextual Orchestrator currently coordinates model +calls and evidence; it does not own domain scientific arithmetic. Adding +unrelated kernels here would blur validation, numerical parity, and data +ownership. + +## Considered alternatives + +- implement all ecosystem arithmetic in Python here: convenient but duplicates + domain ownership and weakens performance/parity authority; +- add GPU code opportunistically per feature: fragments CPU/GPU semantics; +- call opaque external calculations without a contract: hard to verify; +- keep domain arithmetic with its owning service and define a Rust-first rule + only if this service gains such ownership: selected. + +## Decision + +Scientific arithmetic remains owned by the domain service with the relevant +data, validation, and product contract. If Contextual Orchestrator later owns a +new mathematical kernel, its decision ADR must evaluate Rust-first CPU +implementation, optional GPU acceleration, common test vectors, precision and +overflow behavior, deterministic fallback, profiling, and FFI failure isolation. +Orchestration policy and ordinary I/O logic remain Python unless evidence +justifies a different boundary. + +## Consequences + +This repository does not create speculative Rust/GPU code or claim numerical +capability it does not own. Future kernels face a clear evidence threshold. + +## Failure and recovery + +An external domain calculation fails under its adapter contract and cannot be +silently replaced by an approximate model answer. A future accelerator failure +must fall back to a parity-verified CPU path or fail explicitly. + +## Security, privacy, and governance impact + +Domain data stays with its authorized owner. FFI and accelerator boundaries +require memory-safety, input bounds, dependency provenance, and payload +minimization before adoption. + +## Compatibility and migration + +No current runtime migration is required. A future ownership transfer needs a +versioned interface, canonical test vectors, dual-run comparison, rollout, and +rollback. + +## Verification and acceptance + +Future acceptance requires cross-language and CPU/GPU parity, edge/overflow and +property tests, reproducible benchmarks on named hardware, profiler evidence, +fallback behavior, and 100% owned wrapper/error-path coverage. + +## Rollback and supersession + +Disable the new adapter or accelerator and return to the last verified domain +implementation. Supersession requires a new ownership map and numerical +validation plan. + +## References + +NIST SP 800-218 and NIST SP 800-218A. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/0015-provider-egress-response-trust.md b/docs/adr/0015-provider-egress-response-trust.md new file mode 100644 index 000000000..89a502741 --- /dev/null +++ b/docs/adr/0015-provider-egress-response-trust.md @@ -0,0 +1,92 @@ +# ADR-0015: Provider egress and response trust + +## Status + +`active_pr` — PR #96 contains the complete candidate implementation; protected +main retains only the controls explicitly documented there. + +## Context and decision drivers + +Provider configuration can be attacker-influenced or stale, and the provider +receives both a credential and purpose-bound request content. Scheme and +hostname validation do not prevent DNS rebinding, connection-to-validation +drift, ambient proxy use, credential-forwarding redirects, ambiguous HTTP +framing, or resource exhaustion through JSON, JSONL, and SSE responses. The +boundary must remain provider-neutral and independently usable without assuming +an external egress proxy. + +## Considered alternatives + +- trust operator-entered provider URLs: insufficient against mistakes, + compromised configuration, and DNS changes; +- validate URL and DNS, then use the default opener: socket selection, ambient + proxies, and redirects can escape the validated identity; +- require a central proxy: useful as defense in depth, but it breaks standalone + authority and does not validate response semantics; +- retain DNS validation through connection establishment and own bounded + response parsing: selected. + +## Decision + +Non-mock egress requires HTTPS, an optional explicit host allowlist, and only +globally routable resolved addresses. The accepted transport pins a validated +address through connection establishment while preserving the original +hostname for TLS SNI and certificate verification. It ignores ambient proxy +configuration, rejects redirects, constrains timeouts/retries, and bounds header, +body, chunk, cumulative SSE, output-token, and batch-response resources. + +The response boundary rejects conflicting length/transfer framing, invalid or +duplicate JSON keys, non-finite numbers, malformed/truncated UTF-8 and SSE, and +completion without the required terminal state. Errors are bounded and redacted. +None of these active-PR behaviors is described as shipped before protected +merge. + +## Consequences + +Some provider extensions and enterprise proxy assumptions require a reviewed +adapter instead of silent compatibility. The transport surface is larger, but +credentials, memory, and parser state gain one testable authority. + +## Failure and recovery + +Private/non-global resolution, pin drift, TLS failure, redirect, proxy attempt, +invalid framing, excessive bytes/tokens, malformed content, or exhausted retry +budget fails closed. Only explicitly classified timeout, throttle, unavailable, +or eligible 5xx failures may retry or fail over. Recovery disables the affected +provider or restores the last accepted transport; it never allows a private +destination or ambient proxy as a shortcut. + +## Security, privacy, and governance impact + +This boundary addresses SSRF, DNS rebinding, credential exfiltration, redirect +confusion, response smuggling, decompression/resource exhaustion, parser +differentials, and sensitive-error leakage. Provider output remains untrusted +and cannot grant tool, host, review, or credential authority. + +## Compatibility and migration + +`mock://` remains networkless. Compatible providers must offer direct HTTPS +and the documented response subset. Rollout is adapter-local and can be +reverted without rewriting workflow evidence. Deployment-specific egress +controls add defense in depth but do not replace the library boundary. + +## Verification and acceptance + +Acceptance includes DNS rebinding and IPv4/IPv6 tests; allowlist and non-global +rejection; TLS hostname/certificate identity; proxy and redirect rejection; +credential revocation; retry, failover, circuit, and cleanup; framing and +cumulative resource bounds; strict JSON/JSONL/SSE/UTF-8; error redaction; +property/fuzz tests; Semgrep/CodeQL; exact-head coverage/docstrings; and +qualifying review. + +## Rollback and supersession + +Rollback disables affected providers and restores the last protected +implementation while retaining safe global-address validation. Supersession +requires equal or stronger address, connection, TLS, credential, proxy, +redirect, framing, resource, parser, compatibility, and recovery evidence. + +## References + +Bray (2017), Fielding et al. (2022), Rescorla (2018), and OWASP Foundation +(n.d.). Full APA 7 entries are in [the reference index](../REFERENCES.md). diff --git a/docs/adr/0016-complete-coverage-docstrings.md b/docs/adr/0016-complete-coverage-docstrings.md new file mode 100644 index 000000000..06099ae40 --- /dev/null +++ b/docs/adr/0016-complete-coverage-docstrings.md @@ -0,0 +1,87 @@ +# ADR-0016: Complete production coverage and public docstrings + +## Status + +`accepted_architecture` + +## Context and decision drivers + +Coverage evidence can appear complete while omitting owned modules, branches, +functions, package-import paths, or public API explanation. Percentage chasing +can also hide real 4xx/5xx and state-transition defects exposed by realistic +tests. Commercial and acquisition evidence needs exact source identity and +beginner-readable contracts, not a threshold detached from behavior. + +## Considered alternatives + +- accept a lower repository-wide percentage: leaves unclassified product risk; +- exclude difficult or optional production files: can hide real behavior; +- add no-op line execution: raises a number without proving a contract; +- require complete owned statement, branch, function, line, package, and public + docstring evidence with realistic tests: selected. + +## Decision + +Release acceptance requires exact 100% owned production statement, branch, +function, and line coverage where the selected tooling reports each dimension. +Statement and branch coverage remain mandatory in the current gate; unavailable +function or line metrics must be recorded as absent evidence rather than +inferred from another dimension. Every public class, method, and function has a beginner-readable +docstring. The owned-source manifest and checked-out revision are evidence, so a +synthetic merge, predecessor head, stale source tree, skipped-required check, or +status alone is not contributor-head success. + +Tests exercise observable contracts. When a coverage test exposes a real +HTTP/state/provider defect, the defect receives RCA and a failing regression +before the smallest production repair. Structurally unreachable code is removed +or its invariant is documented; it is not excluded to preserve a percentage. + +## Consequences + +Every production branch carries a verification and documentation cost. Reports +are more defensible, but complete coverage remains necessary rather than +sufficient: security scans, fuzz, packaging, provenance, independent review, +and protected-main acceptance remain separate gates. + +## Failure and recovery + +Any missed statement/branch/public docstring, source-tree mismatch, package +build/install/import failure, or required optional-path gap blocks coverage +acceptance. RCA distinguishes a bad assumption, test gap, product defect, +unreachable guard, and infrastructure failure. Recovery adds the real contract +test/docstring or reverts the behavior; it never lowers thresholds or adds a +blanket exclusion. + +## Security, privacy, and governance impact + +Security boundaries receive adversarial, property, and fuzz evidence in +addition to deterministic paths. Fixtures contain no live secrets and coverage +artifacts do not include provider credentials or private reasoning. Coverage +does not impersonate independent approval. + +## Compatibility and migration + +Tools may change, but the owned-source set, exact revision, branch semantics, +function/line evidence, package smoke, and public API contract remain explicit. +Optional adapters require executable evidence or a clearly non-release status. + +## Verification and acceptance + +Run focused regressions, the full functional/integration suite, branch-enabled +coverage over the owned production manifest, public-docstring inspection, +package build/install/import isolation, property/Atheris seams, security gates, +and documentation fitness. Classify every workflow by the commit actually +checked out. + +## Rollback and supersession + +Rollback reverts the production change or supplies the missing real test and +docstring. No rollback weakens the threshold or hides behavior. A stronger +evidence system may supersede this ADR only if it preserves exact source +identity and complete statement, branch, function, line, package, and public +contract proof. + +## References + +NIST SP 800-218 and NIST SP 800-218A. See +[the reference index](../REFERENCES.md). diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 000000000..7b6975de5 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,44 @@ +# Architecture Decision Records + +ADRs record durable product and technical choices. A PR body or conversation is +evidence, not a decision authority. Status follows [the documentation index](../README.md), +and an active-PR decision is never described as shipped. + +| ADR | Decision | Status | +|---|---|---| +| [ADR-0001](0001-route-conduct-test-time-compute.md) | Route versus conduct test-time-compute allocation | `implemented_on_protected_main` | +| [ADR-0002](0002-provider-neutral-transport-trust.md) | Provider-neutral OpenAI-compatible boundary | `accepted_architecture` | +| [ADR-0003](0003-workflow-access-and-reasoning-control.md) | Workflow decomposition, access lists, recursion, and role effort | `accepted_architecture` | +| [ADR-0004](0004-kv-credential-bootstrap.md) | KV credentials and environment-bootstrap-only transport | `implemented_on_protected_main` | +| [ADR-0005](0005-sync-batch-pg-llm-batch.md) | Sync/batch routing and pg-llm-batch integration | `implemented_on_protected_main` | +| [ADR-0006](0006-honest-cost-and-benchmark-evidence.md) | Honest cost/evidence attribution and comparable-budget evaluation | `accepted_architecture` | +| [ADR-0007](0007-free-first-fallback.md) | Free-first fallback and provider-failure semantics | `active_pr` | +| [ADR-0008](0008-state-persistence-and-retention.md) | State/audit persistence and retention | `accepted_architecture` | +| [ADR-0009](0009-purpose-bound-pii-protection.md) | Purpose-bound PII protection without destructive masking | `accepted_architecture` | +| [ADR-0010](0010-independent-review-and-evidence.md) | Independent automated-review identity and evidence separation | `accepted_architecture` | +| [ADR-0011](0011-release-coverage-and-provenance.md) | Release/provenance/SBOM acceptance | `accepted_architecture` | +| [ADR-0012](0012-standalone-and-cwl-boundary.md) | Standalone versus CWL modular authority | `implemented_on_protected_main` | +| [ADR-0013](0013-database-naming-and-migration.md) | Database naming and migration discipline | `accepted_architecture` | +| [ADR-0014](0014-scientific-computation-ownership.md) | Scientific-computation ownership | `out_of_scope` | +| [ADR-0015](0015-provider-egress-response-trust.md) | DNS-pinned egress, redirect/proxy rejection, and bounded response trust | `active_pr` | +| [ADR-0016](0016-complete-coverage-docstrings.md) | Complete production coverage and public-docstring evidence | `accepted_architecture` | + +## Minimum decision coverage + +The set separately records all required decisions: route/conduct allocation; +provider-neutral compatibility; workflow decomposition/access/effort; KV +credentials; provider transport trust; sync/batch; honest cost and evaluation; +free-first fallback; standalone/CWL authority; state/audit retention; +independent review identities; complete coverage/docstrings; release +provenance/SBOM; and purpose-bound PII. Database migration and scientific +ownership remain additional explicit decisions rather than being hidden in +unrelated ADRs. + +## Lifecycle + +Create a new ADR when drivers, ownership, compatibility, or recovery changes. +Do not rewrite accepted history to hide a reversal: mark it `superseded`, link +the replacement, and preserve migration and rollback evidence. Each ADR covers +context and drivers, alternatives, decision, consequences, failure/recovery, +security/privacy/governance, compatibility/migration, verification/acceptance, +rollback, and supersession. diff --git a/docs/analytics_spec.md b/docs/analytics_spec.md index f77d81700..213cca788 100644 --- a/docs/analytics_spec.md +++ b/docs/analytics_spec.md @@ -1,5 +1,8 @@ # Analytics Spec +**Document state:** `implemented_on_protected_main` for the listed local runtime +reports; production telemetry and warehouse-backed evidence remain planned. + ## Measurement Context This repository does not include production telemetry, event logs, or a @@ -10,44 +13,44 @@ dashboard claims real usage. In short: this spec provides proposed definitions and source requirements, not measured product results. -The stdlib prototype now exposes `/api/v1/analytics_snapshots/latest` as a +The standalone runtime exposes `/api/v1/analytics_snapshots/latest` as a local runtime snapshot. It measures only in-memory events and workflow records from the current process, so it is source-backed for smoke tests and pilot readiness checks, but it is still not production telemetry or a warehouse-backed dashboard. -The prototype also exposes `/api/v1/sales_readiness/latest`. That endpoint +The standalone runtime also exposes `/api/v1/sales_readiness/latest`. That endpoint turns the local snapshot, admin state, HTTP security profile, locale bundles, and provider configuration into explicit pass/warn/fail criteria for enterprise pilot review. It is a readiness gate for a sellable pilot, not a production compliance certificate or proof of real customer usage. -The prototype also exposes `/api/v1/commercial_readiness/latest`. That endpoint +The standalone runtime also exposes `/api/v1/commercial_readiness/latest`. That endpoint rolls product, security, operations, audit, documentation, support, localization, and value-case evidence into a KRW 2,000,000,000 buyer due-diligence gate. It is measured only as local due-diligence evidence and is not a valuation guarantee, purchase commitment, or production compliance certificate. -The prototype also exposes `/api/v1/commercial_launch_readiness/latest`. That +The standalone runtime also exposes `/api/v1/commercial_launch_readiness/latest`. That endpoint packages the GTM packet, runtime path, acceptance tests, operator runbook, admin evidence, analytics truthfulness, Figma artifacts, review policy, and packaging decision while keeping buyer environment, production telemetry, and commercial signature inputs as explicit warnings. -The prototype also exposes `/api/v1/commercial_completion_scorecards/latest`. +The standalone runtime also exposes `/api/v1/commercial_completion_scorecards/latest`. That endpoint converts the KRW 2,000,000,000 completion scorecard into runtime evidence across Product Design, Figma, Superpowers, Ponytail, Data Analytics, runtime readiness, verification, review policy, packaging, and external follow-ups. -The prototype also exposes +The standalone runtime also exposes `/api/v1/commercial_buyer_acceptance_workflows/latest`. That endpoint converts the buyer acceptance runbook into owner-scoped runtime workflow evidence while keeping production and buyer-specific follow-ups as warnings, not measured results. -The prototype also exposes `/api/v1/commercial_demo_scenarios/latest`. That +The standalone runtime also exposes `/api/v1/commercial_demo_scenarios/latest`. That endpoint packages the buyer demo script as local runtime evidence across compatible API smoke, workflow trace, access-list evidence, evaluation replay, admin readiness, metric truthfulness, Figma stakeholder review, buyer acceptance, @@ -55,14 +58,14 @@ review-process policy, and packaging decision. Production telemetry, ROI, legal, security questionnaire, and support-plan inputs remain proposed or buyer-specific warnings until supplied. -The prototype also exposes `/api/v1/commercial_proposal_packets/latest`. That +The standalone runtime also exposes `/api/v1/commercial_proposal_packets/latest`. That endpoint packages the buyer proposal review artifact as local runtime evidence across completion, demo, acceptance, value, security, contract, onboarding, operations, analytics truthfulness, Figma stakeholder review, review-process policy, and packaging decision. Pricing, legal, ROI, production, support, and signature inputs remain proposed or buyer-specific warnings until supplied. -The prototype also exposes +The standalone runtime also exposes `/api/v1/commercial_purchase_approval_packets/latest`. That endpoint packages the buyer purchase approval artifact as local runtime evidence across proposal, close, procurement, contract, value, security, onboarding, operations, analytics @@ -71,7 +74,7 @@ decision. Buyer signature authority, budget approval, purchase order, finance authority, and go-live authorization remain proposed or buyer-specific warnings until supplied. -The prototype also exposes `/api/v1/commercial_due_diligence_rooms/latest`. That +The standalone runtime also exposes `/api/v1/commercial_due_diligence_rooms/latest`. That endpoint packages the buyer due diligence room as local runtime evidence across purchase approval, runtime API evidence, admin trace/access evidence, security, commercial terms, value analytics, implementation readiness, Figma stakeholder @@ -79,7 +82,7 @@ review, review-process policy, and packaging decision. Buyer authority documents, production telemetry, and third-party attestations remain proposed or buyer-specific warnings until supplied. -The prototype also exposes +The standalone runtime also exposes `/api/v1/commercial_investment_committee_memos/latest`. That endpoint packages the executive investment committee memo as local runtime evidence across due diligence, purchase approval, financial case, risk/security, commercial terms, @@ -88,7 +91,7 @@ packaging decision. Buyer final authority, production telemetry, and external attestations remain proposed or buyer-specific warnings until supplied. Decision supported: decide whether Contextual Orchestrator is ready to move from -lab prototype to enterprise pilot while preserving traceability, compliance +local evaluation to enterprise pilot while preserving traceability, compliance evidence, and API compatibility. Review cadence: weekly during pilot, then monthly for operating review. diff --git a/docs/architecture.md b/docs/architecture.md index c0f63a81e..10234a15e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,15 +1,21 @@ # Architecture Notes +**Document state:** `superseded` as system authority; retained as a research +mapping. See [root architecture](../ARCHITECTURE.md) for current implementation +and trust boundaries. + ## Sources Read - Sakana AI launch article, "Sakana Fugu: One Model to Command Them All" (June 22, 2026): https://sakana.ai/fugu-release/ -- Sakana Fugu Technical Report: https://github.com/SakanaAI/fugu/blob/main/Fugu_technical_report.pdf +- Sakana Fugu Technical Report: https://doi.org/10.48550/arXiv.2606.21228 - TRINITY: An Evolved LLM Coordinator: https://arxiv.org/abs/2512.04695 - Learning to Orchestrate Agents in Natural Language with the Conductor: https://arxiv.org/abs/2512.04388 ## What The Architecture Is -The public shape is a single model API. The internal shape is a model pool plus a learned coordinator that decides when to answer directly, when to delegate, how much context each worker receives, when to verify, and how to synthesize the final answer. +The cited papers' public shape is a single model API. Their internal shape can +include learned coordination. Protected main instead uses deterministic policy; +it does not claim a learned coordinator. The useful split is quality-latency, not separate products: @@ -31,9 +37,10 @@ The Fugu report combines these ideas into production constraints: This repository implements the interface and control plane, not the trained coordinator. -- `contextual_orchestrator.orchestrator.Agent`: one configured worker model. -- `Orchestrator.route_once`: the low-latency routing path. -- `Orchestrator.conduct`: the workflow path with planner, worker, verifier, and synthesizer steps. +- `contextual_orchestrator.orchestrator.ModelAgent`: one configured worker model. +- `TaskOrchestrator.route_once`: the low-latency routing path. +- `TaskOrchestrator.conduct`: the workflow path with planner, worker, verifier, and synthesizer steps. +- `CostRoutingCoordinator`: the separate sync-versus-batch and ledger layer. - `WorkflowStep.access`: Conductor-style visibility control. - `ModelClient`: OpenAI-compatible HTTP client, with `mock://` for local checks. - `contextual_orchestrator.server`: small `/v1/chat/completions` HTTP server. diff --git a/docs/doctoring/security-disclosure-lifecycle.md b/docs/doctoring/security-disclosure-lifecycle.md new file mode 100644 index 000000000..de2c9d9fa --- /dev/null +++ b/docs/doctoring/security-disclosure-lifecycle.md @@ -0,0 +1,68 @@ +# Security disclosure lifecycle doctoring + +## Decision + +`SECURITY.md` defines a bounded coordinated vulnerability disclosure and handling lifecycle rather than only a reporting address. The policy separates communication targets from remediation guarantees, keeps unpatched details private, preserves repository release gates for security fixes, and identifies exact released-version evidence as the authoritative remediation boundary. + +The policy is intentionally repository-local. It does not authorize testing of third-party providers or other ContextualWisdomLab repositories and does not convert access to public endpoints into permission for destructive, privacy-invasive, or high-volume testing. + +## Primary evidence reviewed + +Evidence was rechecked on 2026-08-08 against primary publisher documentation. + +### ISO/IEC 29147:2018 + +ISO identifies ISO/IEC 29147:2018, *Information technology — Security techniques — Vulnerability disclosure*, as the current published second edition. ISO states that the standard provides requirements and recommendations for receiving reports about potential vulnerabilities and disclosing remediation information. ISO's catalogue states that this edition was last reviewed and confirmed in 2024 and remains current. This supports a documented private-reporting path, disclosure coordination, affected/remediated-version communication, and explicit policy boundaries. + +### ISO/IEC 30111:2019 + +ISO identifies ISO/IEC 30111:2019, *Information technology — Security techniques — Vulnerability handling processes*, as the current published second edition. ISO states that it covers processing and remediating reported potential vulnerabilities. ISO's catalogue states that this edition was reviewed and confirmed in 2025 and remains current. This supports the receive → validate/scope → remediate/verify → coordinate release → publish → learn lifecycle in `SECURITY.md`. + +### GitHub vulnerability reporting and repository advisories + +GitHub's current documentation describes GitHub private vulnerability reporting as a structured private channel for public repositories when the feature is enabled. GitHub also documents the repository security advisory workflow as a private collaboration mechanism for discussing, fixing, and publishing vulnerability information. GitHub recommends that `SECURITY.md` explain supported versions and reporting instructions. These sources support the repository's primary reporting URL, public-issue fallback that contains no exploit detail, private remediation collaboration, reporter credit, and advisory publication boundary. + +### NIST SSDF status + +NIST SP 800-218 Rev. 1, Secure Software Development Framework Version 1.2, is currently an **Initial Public Draft**, published 2025-12-17; its public comment period has closed. The official NIST publication metadata lists Harold Booth, Michael Ogata, Karen Kent, Murugiah Souppaya, and Donna Dodson as the authors and identifies DOI `10.6028/NIST.SP.800-218r1.ipd`. It is therefore contextual acquisition and secure-development evidence, not a finalized normative requirement. NIST describes SSDF as a common set of practices for reducing vulnerabilities and notes its usefulness in supplier/acquirer communication. The repository policy uses that evidence only to reinforce the need for verified remediation and release evidence; ISO/IEC 29147 and ISO/IEC 30111 remain the primary disclosure/handling standards cited by the policy. + +## Repository contract + +The bounded buyer-visible contract is aligned with the canonical +[`docs/RELEASE_GUIDE.md`](../RELEASE_GUIDE.md): + +1. No stable release currently exists, and `main` is not a supported release. When a stable release is published, its supported version or release line becomes the default support boundary; an advisory may explicitly include additional supported release lines. +2. `main`, development branches, archived artifacts, forks, and historical tags are not automatically represented as supported releases. +3. Private vulnerability reporting is the preferred channel. Its enablement and security-notification recipients are stable-release admission checks. If unavailable, a public issue may request a secure channel but must not disclose exploit details, secrets, personal data, or unreleased vulnerability details; release authorization remains blocked until a monitored alternative private contact is documented. +4. The five-business-day acknowledgement target is a communication objective and **not a remediation SLA**. +5. Security remediation follows normal exact-head security, coverage, provenance, independent-review, branch-protection, packaging, and release-acceptance gates; urgency does not create a bypass. +6. A GitHub Security Advisory should identify affected and patched versions and may request a CVE when warranted and available. +7. Reporter credit is opt-in/appropriate to the coordinated-disclosure context and may be declined. +8. Release evidence for the exact integrated revision fails closed: queued, pending, skipped-required, cancelled, failed, absent, stale-head, predecessor-head, author-only, status-only, synthetic-merge-only, rate-limited, or infrastructure-only evidence is not passing evidence. +9. The safe-harbor language is bounded good-faith guidance, not authorization against third parties or systems/data the researcher does not control. + +## Verification + +`tests/test_repository_security_metadata.py::test_security_policy_documents_coordinated_disclosure_lifecycle` locks the buyer-visible policy vocabulary and this evidence receipt. The test is deliberately documentation-focused: it prevents future edits from silently deleting the supported-version boundary, lifecycle, non-SLA qualification, advisory/CVE path, reporter-credit expectation, public-reporting safety rule, or standards provenance. + +This slice does not change production runtime code, provider behavior, credentials, workflows, database objects, or release state. It also does not modify the central `.github` control plane or depend on unmerged central coverage logic. + +## References (APA 7) + +GitHub. (n.d.). *Adding a security policy to your repository*. GitHub Docs. Retrieved August 8, 2026, from https://docs.github.com/en/code-security/how-tos/report-and-fix-vulnerabilities/configure-vulnerability-reporting/add-security-policy + +GitHub. (n.d.). *Coordinated disclosure of security vulnerabilities*. GitHub Docs. Retrieved August 8, 2026, from https://docs.github.com/en/code-security/concepts/vulnerability-reporting-and-management/coordinated-disclosure + +GitHub. (n.d.). *Privately reporting a security vulnerability*. GitHub Docs. Retrieved August 8, 2026, from https://docs.github.com/en/code-security/how-tos/report-and-fix-vulnerabilities/report-privately + +GitHub. (n.d.). *Repository security advisories*. GitHub Docs. Retrieved August 8, 2026, from https://docs.github.com/en/code-security/concepts/vulnerability-reporting-and-management/repository-security-advisories + +International Organization for Standardization. (2018). *ISO/IEC 29147:2018 Information technology—Security techniques—Vulnerability disclosure* (2nd ed.). https://www.iso.org/standard/72311.html + +International Organization for Standardization. (2019). *ISO/IEC 30111:2019 Information technology—Security techniques—Vulnerability handling processes* (2nd ed.). https://www.iso.org/standard/69725.html + +Booth, H., Ogata, M., Kent, K., Souppaya, M., & Dodson, D. (2025). *Secure software development framework (SSDF) version 1.2: Recommendations for mitigating the risk of software vulnerabilities* (NIST SP 800-218 Rev. 1, Initial Public Draft). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-218r1.ipd + +## APA 7 note + +ISO standards and GitHub first-party documentation use organizational authors. The NIST draft uses the individual authors listed in the official publication metadata, with NIST retained as publisher and the publication DOI used as the persistent locator. Retrieval dates are included for GitHub pages because operational documentation can change without a new edition identifier. ISO edition years and NIST publication status are retained as publisher-controlled version evidence. diff --git a/docs/evidence/2026-08-11-documentation-audit.md b/docs/evidence/2026-08-11-documentation-audit.md new file mode 100644 index 000000000..b88597172 --- /dev/null +++ b/docs/evidence/2026-08-11-documentation-audit.md @@ -0,0 +1,151 @@ +# Documentation audit evidence — 2026-08-11 + +**Document state:** `active_pr`; the audit describes protected-main evidence, +while this canonical documentation repair is not shipped until protected merge
+**Audit date:** 2026-08-11 (Asia/Seoul) +**Protected-main revision audited:** `6841b71935e0b7cb98fb52bcb4709cc5100c8d87` + +This file is the only canonical location for volatile audit SHAs and run IDs. +Other architecture documents describe durable contracts. + +## Sufficiency assessment before this documentation repair + +| Artifact family | Prior classification | Evidence | Repair in this branch | +|---|---|---|---| +| PRD | `PARTIAL` | `conductor/product.md`, `docs/product_planning.md`, and user stories described intent but did not provide one status-qualified product contract. | `docs/PRD.md` | +| TRD | `PARTIAL` | API, database, KV, analytics, and security requirements were scattered and mixed target architecture with runtime behavior. | `docs/TRD.md` | +| Architecture | `STALE` | `docs/architecture.md` named `Agent`/`Orchestrator`, omitted current stores/cost/batch surfaces, and described an earlier implementation. | root `ARCHITECTURE.md` | +| ADR | `MISSING` | No status-bearing ADR directory or decision index existed. | `docs/adr/` | +| UML | `MISSING` | No checked-in Mermaid/PlantUML runtime sequences, states, or deployment authority diagram existed. | `docs/UML.md` | +| ERD/data ownership | `PARTIAL` | `docs/database_design.sql` described a normalized target but not actual SQLite/PEP-249/Postgres objects or in-memory/external ownership. | `docs/ERD.md` | +| Threat model | `MISSING` | `SECURITY.md` provided disclosure and scanner policy, not assets, zones, abuse cases, controls, and residual risk. | `docs/THREAT_MODEL.md` | +| Test strategy | `PARTIAL` | Tests and fuzz docs existed, but no exact-head evidence taxonomy or release-wide test contract. | `docs/TEST_STRATEGY.md` | +| Operability/runbook | `PARTIAL` | Commercial packets mentioned gaps; no canonical degraded-mode and recovery authority existed. | `docs/OPERABILITY.md`, `docs/INCIDENT_RUNBOOK.md` | +| Release operations | `PARTIAL` | ADR-0011 defined the decision and gates, but no canonical operator sequence joined exact source identity, artifact provenance, migration/rollback, publication, and protected-main acceptance. | `docs/RELEASE_GUIDE.md` | +| Research/standards | `PARTIAL` | Paper PDFs and an architecture note existed, but APA 7 and current official standards were not indexed together. | `docs/REFERENCES.md` | +| Documentation index | `MISSING` | Buyers and maintainers could not discover which artifact was authoritative. | `docs/README.md` | +| Documentation fitness test | `MISSING` | Existing tests checked selected keywords and could pass while canonical families were absent or stale. | `tests/test_documentation_contract.py` | +| Changelog | `MISSING` on audited main | Protected main had no root `CHANGELOG.md`; PR #96 introduces one on an active stack. | Not duplicated here; resolve through the accepted stack to avoid conflicting authority. | + +The pre-repair set was therefore **not sufficient** for commercial or +acquisition diligence. Volume was not the issue: many buyer packets existed, +but product requirements, implementation truth, decisions, diagrams, and data +ownership were not joined into one status-disciplined graph. + +## Requirement-to-implementation matrix + +| Requirement | Product state | Implementation authority | Decision/docs | Test authority | +|---|---|---|---|---| +| PRD-001 / FR-001 compatible chat surface | `implemented_on_protected_main` | `server.py`, `orchestrator.py`; OpenAPI subset drift is recorded in TRD | PRD, TRD, ADR-0002 | API and passthrough tests | +| PRD-002 / FR-002 route/conduct allocation | `implemented_on_protected_main` | `TaskOrchestrator.complete` | ADR-0001, UML | paper/optimizer tests | +| PRD-003 / FR-003/004 explicit workflow/access | `implemented_on_protected_main` | `WorkflowStep`, conduct/generated planner | ADR-0003, UML | paper/generated-workflow tests | +| PRD-004 / FR-005 reliability/failover | `implemented_on_protected_main` | `ModelClient`, `TaskOrchestrator` circuit state | Architecture, threat model | provider-reliability tests | +| PRD-005 / FR-006 KV credentials | `implemented_on_protected_main` | `credentials.py`, `kv_config.py`, CLI | ADR-0004, UML, threat model | KV credential tests | +| PRD-006 / FR-007 cost attribution | `implemented_on_protected_main` with honesty gaps | two unsynchronized authorities in `orchestrator.py`, `cost_ledger.py`, `cost_router.py` | ADR-0006, ERD | cost-ledger tests plus reconciliation/unknown-price tests required | +| PRD-007 / FR-008 sync/batch | `implemented_on_protected_main` with restart/idempotency gaps | `batch_routing.py`, `cost_router.py` | ADR-0005, UML | batch tests plus restart/replay tests required | +| PRD-008 / FR-009 optional persistence | `implemented_on_protected_main` | `_StateStore`, `_AgentPoolStore`, SQL/KV adapters | ADR-0008, ERD | persistence/agent-pool/ledger tests | +| FR-011 route registry parity | `accepted_architecture` | dispatcher and static OpenAPI currently diverge | TRD, test strategy | shared-registry parity test required | +| FR-012 execution-path evidence parity | `accepted_architecture` | passthrough/streaming bypasses are documented | Architecture, UML, threat model | mode matrix required | +| PRD-006 / FR-013 cost authority | `accepted_architecture` | unknown ledger price is currently zero; SQL price table dormant | ADR-0006, ERD | reconciliation and non-free unknown tests required | +| PRD-007 / FR-014 durable batch identity | `accepted_architecture` | job/idempotency maps are process-local | ADR-0005, operability | restart and replay tests required | +| PRD-009 / SEC-002 strict provider transport | `active_pr` | PR #96 | ADR-0002, ADR-0015, threat model | Evidence remains PR-bound | +| Free-first fallback | `active_pr` | PR #94 | ADR-0007 | Evidence remains PR-bound | +| Adaptive reasoning effort | `active_pr` | PR #99 stacked on #94 | ADR-0003 | Evidence remains PR-bound | +| NIM all-modality benchmark | `active_pr` | PR #90 stacked on #96 | ADR-0006 | Evidence remains PR-bound | +| Local loopback MLX provider and audited model judgment | `active_pr` | PR #109 independently targets protected main | PR #109 planning ADR and tests | Evidence remains PR-bound | +| PRD-010 independent review and release | `accepted_architecture` | GitHub rules/workflows and human governance | ADR-0010, ADR-0011, ADR-0016, test strategy | Exact-head and protected-main evidence | +| Purpose-bound PII handling | `accepted_architecture` | Host plus runtime audience boundaries | ADR-0009, threat model | Privacy/telemetry/trace tests and deployment evidence | + +## Dated open-PR snapshot + +All PR facts below were refetched during this audit. `base tip` is the live +branch ref, not the historical base snapshot stored in PR metadata. The table +records the live state immediately before the evidence commit. Because PR #105 +contains this ledger and PR #104 is stacked on it, publishing this ledger +necessarily advances those two branch refs. Their rows are therefore pre-write +audited identities, not post-write current-head claims. Later acceptance must +refetch both refs rather than treating this dated table as merge authority. + +| PR | Audited contributor head (pre-write) | Base branch → live tip | Draft / mergeable | Observed gate summary | Unresolved threads | +|---:|---|---|---|---|---:| +| #110 | `8607eba46a5dd7773fde211ceedcf70b3855de0d` | `codex/local-llm-benchmark` → `c138d1737b69fec9805253398a41046617a3a7a7` | yes / no | Direct local suite reports 324 tests; default-branch hosted workflows and 100% production coverage are absent on this exact stacked head; CodeRabbit skipped the non-default base; no formal review or qualifying approval | 0 | +| #109 | `ada372df205271c74ad095e898644588c7156075` | `main` → `6841b71935e0b7cb98fb52bcb4709cc5100c8d87` | no / yes | Exact-head Tests, Security, Fuzz, and Security Scan success; merge-tree Semgrep still fails on one `HTTPSConnection` advisory; one COMMENTED security review and no qualifying approval | 0 | +| #108 | `8760993cb8262922a771948845c8dfd2afefb773` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Exact-head Tests, Security, and Fuzz success; 571 tests and 100% statement/branch/docstring evidence; built package identity, license, and SBOM evidence present; no formal review or qualifying approval | 0 | +| #107 | `28088b9fc86d975b43637b7758d25e20d61c5786` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Exact-head Tests, Security, and Fuzz success; no formal review or qualifying approval | 0 | +| #105 | `828ca54f2b96a3bdd7adec24a26c0d8164df47d1` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Exact-head Tests, Security, and Fuzz success; 585 tests and 100% statement/branch/docstring evidence; no formal review or qualifying approval | 0 | +| #104 | `8453f082672d96b564ff2c32d028b11e05d8729f` | `docs/canonical-product-architecture` → `828ca54f2b96a3bdd7adec24a26c0d8164df47d1` | yes / yes | Exact-head Tests, Security, and Fuzz success; 586 tests and 100% statement/branch/docstring evidence; no formal review or qualifying approval | 0 | +| #99 | `2502915a8e90059074167e6306b47148a1d40fdc` | `feat/free-first-model-fallback-policy` → `73ed3a077f88a2f03cf734f1067bee2dcce2467f` | yes / yes | Exact-head quality, Tests, Security, and Fuzz success; no formal approval | 0 | +| #94 | `73ed3a077f88a2f03cf734f1067bee2dcce2467f` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Exact-head quality, Tests, Security, and Fuzz success; prior OpenCode findings dismissed | 0 | +| #96 | `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | `main` → `6841b71935e0b7cb98fb52bcb4709cc5100c8d87` | yes / yes | Exact-head Tests, Security, and Fuzz success; Security Scan/Semgrep are integration evidence; central prerequisite and qualifying approval absent | 0 | +| #90 | `26f8d8dc5634f0371fad0801056e9a3450c78bff` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / no | Exact-head Tests, Security, and Fuzz success; one addressed thread remains open pending every required gate | 1 | +| #82 | `f56337f4cc9a170ba999b82419666be5027497d1` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / no | Pre-refresh Tests, Security, Fuzz, and Security Scan success; Semgrep failure; evidence does not transfer | 0 | +| #75 | `8bc91f370eefc2a907170303ae27315ec567bf74` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / no | Named workflows success; stale stack awaits post-#96 reconstruction | 0 | +| #66 | `e7020795c6c5cbaac884dbcee3e0a37c409ab360` | `claude/contextualwisdomlab-audit-governance-fb7470` → `8bc91f370eefc2a907170303ae27315ec567bf74` | yes / no | Named workflows success; downstream stack awaits accepted #75 result | 0 | +| #71 | `2f4ec9fed753927d1ebc83638db68683736e6fad` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Tests, Security, Fuzz, and Security Scan success; stale-base Semgrep failure | 0 | +| #69 | `e0b3bcf31b42e284e8d0519751cfa0e775cfa32b` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Tests, Security, Fuzz, and Security Scan success; stale-base Semgrep failure | 0 | +| #83 | `fa3a30bda3b3209025d55c5526a037f3086f0f07` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Tests, Security, Fuzz, and Security Scan success; stale-base Semgrep failure | 0 | +| #63 | `dd4e62b46fbc651a6696cb04438751122e161d8c` | `fix/atheris-interpreter-lock` → `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | yes / yes | Tests, Security, Fuzz, and Security Scan success; stale-base Semgrep failure | 0 | + +16 of 17 open PRs were Draft in the snapshot. PR #109 was Ready but remained +ineligible for immediate protected merge, as was every Draft PR. A successful +workflow name or CodeRabbit status was not +promoted into independent approval or exact-head success. Live ruleset detail +was not returned by the connector used for this audit, so the repository's +required-context decision remains GitHub's protected merge authority rather +than a reconstructed list. + +## Dependency order from live refs + +```mermaid +flowchart TB + centralMain["read-only .github protected main"] --> central906[".github PR #906 direct redaction repair"] + central906 --> central929[".github PR #929 JSON repair: test-only RED"] + central906 --> central907[".github issue #907 wrapper repair: no completing PR"] + central929 -. protected integration required .-> pr96 + central907 -. protected integration required .-> pr96 + main["protected main"] --> pr96["PR #96 provider and Atheris boundary"] + main --> pr109["PR #109 local MLX and audited judgment"] + pr109 --> pr110["PR #110 Keyverse tenant authorization"] + pr96 --> pr107["PR #107 CodeQL action update"] + pr96 --> pr108["PR #108 package identity and license authority"] + pr96 --> pr105["PR #105 canonical documentation"] + pr105 --> pr104["PR #104 disclosure lifecycle"] + pr96 --> pr82["PR #82 pip bootstrap"] + pr96 --> pr90["PR #90 NIM benchmark"] + pr96 --> pr94["PR #94 free-first fallback"] + pr94 --> pr99["PR #99 adaptive reasoning"] + pr96 --> pr75["PR #75 coverage and latent fixes"] + pr75 --> pr66["PR #66 embeddings and KV bootstrap"] +``` + +The central dependency is read-only here. At audit time, #906 had ten green +exact-head workflows but no formal review or qualifying approval. Its own scope +left wrapper operands (#907) and multiline/duplicate-key JSON (#908) +unresolved. Stacked #929 held the #908 test-first contract at a **test-only +RED** head with no production repair or associated workflow run, while #907 +had **no current completing PR**. None of those states is protected integration +or transferable acceptance evidence for PR #96. + +PR #109 independently targets protected `main`; its current head does not include +#96, and its merge-tree Semgrep gate remains nonpassing. It cannot merge on +status-only review evidence. + +PR #96 supersedes closed-unmerged #76. PR #82 must remain Draft until #96 has +one accepted stable head or protected merge, then preserve only its unique pip +bootstrap intent on the accepted base and reacquire all evidence. + +## Open issues at audit time + +| Issue | Meaning | Related path | +|---:|---|---| +| #95 | Portable Atheris lock | PR #96 | +| #103 | Fail-closed release readiness on exact-head review/check evidence | ADR-0010 and product backlog | +| #102 | Race equivalent model-group endpoints by first valid completion | Planned reliability/product slice | +| #86 | Evidence-grade NIM discovery and cost-quality benchmark | PR #90 | + +## Documentation maintenance rule + +Any change to a mapped requirement must update its PRD/TRD status, ADR if the +decision changes, UML/ERD if control or data flow changes, and this matrix. The +documentation contract test verifies structure; reviewers still verify factual +truth against live code and protected evidence. diff --git a/docs/evidence/2026-08-12-continuation-audit.md b/docs/evidence/2026-08-12-continuation-audit.md new file mode 100644 index 000000000..7d2e98d2c --- /dev/null +++ b/docs/evidence/2026-08-12-continuation-audit.md @@ -0,0 +1,102 @@ +# Continuation evidence audit — 2026-08-12 + +**Document state:** `active_pr`; this appendix records collected evidence and +does not make the documentation branch or any other pull request shipped
+**Audit date:** 2026-08-12 (Asia/Seoul)
+**Protected-main revision observed:** +`6841b71935e0b7cb98fb52bcb4709cc5100c8d87` + +This continuation preserves the 2026-08-11 audit as immutable history and +records the next live evidence collection. Every identity below becomes +`historical` as soon as its branch, protected base, workflow attempt, review, +or ruleset changes. + +## Evidence identity rules + +- `exact_head_success` means the repository workflow evidence explicitly bound + the relevant checkout to the contributor commit. A green workflow merely + associated with a commit is not promoted when its job checkout is unknown or + used a synthetic merge. +- `blocked` means a required authority is missing or non-passing. It does not + mean the implementation is defective. +- `absent` means no qualifying evidence was observed; a status, reaction, + author review, dismissed review, or model review is not substituted. +- `historical` evidence remains useful for diagnosis but cannot authorize a + later head, merge, or release. + +## Protected and dependency authority + +| Authority | Exact revision | Collected evidence | Decision | +|---|---|---|---| +| contextual-orchestrator protected `main` | `6841b71935e0b7cb98fb52bcb4709cc5100c8d87` | Protected ref remained unchanged during collection. | Shipped authority; release acceptance gaps below remain open. | +| central `.github` protected `main` | `6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba` | Read-only dependency; none of the three current repair heads was integrated. | Central working-branch evidence is not repository authority. | +| central PR #937 | `67d834f510fe044dd9d53cd4f4b9783353e303bd` | Eleven terminal-success workflows, zero unresolved threads, and one OpenCode `APPROVED` model review. | `blocked`: the model review is not a qualifying independent human approval and the PR remains open. | +| central PR #939 | `ac5665148bb113f92e97d2fc49a729bca2f050b5` | Nine terminal-success workflows, zero unresolved threads, and no formal review. | `blocked`: review and protected integration are `absent`. | +| central PR #943 | `601b254f3a8ea4cc593e7089d6baeadd9d8d3ee4` | Nine terminal-success workflows, zero unresolved threads, and no formal review. | `blocked`: review and protected integration are `absent`. | + +The central repository was inspected only. No central branch, comment, thread, +review, workflow, or merge state was mutated by this repository loop. + +## Open pull-request continuation snapshot + +The branch ref and protected base were resolved independently where the PR was +evaluated. Workflow summaries below distinguish explicit contributor-checkout +evidence from head-associated records that were not independently promoted. +Every open PR remained ineligible for immediate protected merge because a +qualifying independent non-author approval was `absent`, the PR was Draft or +stack-blocked, or a current changes-requested review remained effective. + +The PR #105 row is a pre-write snapshot. Publishing this appendix advances the +PR #105 branch beyond the recorded row. The row is not current-head evidence +after publication and must be refetched. + +| PR | Contributor head | Collected evidence | Continuation decision | +|---:|---|---|---| +| #63 | `292c87da7bdf3d538710a28f9d94802767ff15f7` | Tests, Security, and Fuzz terminal-success; OpenCode changes requested; author approval only; zero unresolved threads. | `blocked`; downstream dependency update stays Draft and author approval is non-qualifying. | +| #66 | `e7020795c6c5cbaac884dbcee3e0a37c409ab360` | Five terminal-success workflow records; OpenCode changes requested; zero unresolved threads. | `blocked`; reconstruct after the accepted #96/#75 line and regenerate evidence. | +| #69 | `27432737188ea43a0e81ecd279d66cb16c3a00ab` | Tests, Security, and Fuzz terminal-success; OpenCode changes requested; author approval only. | `blocked`; stack order remains #96, #82, then #69. | +| #71 | `1c058275259daad1cbcce96683b0e44137d99a38` | Tests, Security, and Fuzz terminal-success; OpenCode changes requested; author approval only. | `blocked`; dependency work cannot precede #96 integration. | +| #75 | `8bc91f370eefc2a907170303ae27315ec567bf74` | Five terminal-success workflow records; current OpenCode changes requested; zero unresolved threads. | `blocked`; the review request remains non-passing and the stack is stale. | +| #82 | `9341bac45484bdc53b2f8813baac85ee118b176b` | Tests, Security, and Fuzz terminal-success; OpenCode changes requested; zero unresolved threads. | `blocked`; remains Draft behind #96 and must be rebuilt or refreshed after protected integration. | +| #83 | `b5780716c07fc16391e3a525917786ead065dc60` | Tests, Security, and Fuzz terminal-success; OpenCode changes requested; author approval only. | `blocked`; dependency work cannot precede #96 integration. | +| #90 | `26f8d8dc5634f0371fad0801056e9a3450c78bff` | Tests, Security, and Fuzz terminal-success; sixteen COMMENTED reviews; zero currently unresolved threads. | `blocked`; benchmark evidence stays `active_pr` and must be reconciled after #96. | +| #94 | `73ed3a077f88a2f03cf734f1067bee2dcce2467f` | Quality, Tests, Security, and Fuzz terminal-success; three dismissed predecessor reviews; zero unresolved threads. | `blocked`; dismissed review evidence is `historical`, not approval. | +| #96 | `3703d0da9823b8258a0be94f1801aa5d61bfad9f` | Tests, Security, and Fuzz are `exact_head_success`; Security Scan and Semgrep include integration identity; zero unresolved threads. | `blocked`; required automated-review authority, qualifying approval, and protected central acceptance are `absent`. | +| #99 | `b80a30eb4bf9cc0f7c77c58e4d429c9d9fe268db` | Quality, Tests, Security, and Fuzz terminal-success; no formal review; zero unresolved threads. | `blocked`; stacked fallback dependency and qualifying approval are unresolved. | +| #104 | `2b7bf1a8bb8aa361bd1e9ec9038547b3807a730f` | Tests, Security, and Fuzz exact-head records; one CodeRabbit COMMENTED review; zero unresolved threads. | `blocked`; remains Draft behind #105 and #96. | +| #105 | `5543d1b493ceb9dbac485e10347820929d6bee92` | Tests, Security, and Fuzz exact-head records; one predecessor-head CodeRabbit COMMENTED review; zero unresolved threads. | `blocked`; documentation is `active_pr`, not protected-main authority. | +| #107 | `28088b9fc86d975b43637b7758d25e20d61c5786` | Tests, Security, and Fuzz terminal-success; no formal review; zero unresolved threads. | `blocked`; remains a bounded dependency update behind #96. | +| #108 | `1e2600ee442a4894c4ad023f57efa13acfc93a87` | Tests, Security, and Fuzz terminal-success; COMMENTED reviews only; zero unresolved threads. | `blocked`; package metadata remains on the unintegrated security base. | +| #109 | `216177f2c3524a145b24e6b9eafa3e8ca86306f5` | Five terminal-success workflow records and COMMENTED reviews; zero unresolved threads; direct-head coverage acceptance remains non-passing. | `blocked`; workflow association is not substituted for the missing 100% direct-head acceptance and approval. | +| #110 | `5a065bb44b4b7296f68ec992b04ab36b85d90e0e` | No workflow run or formal review was observed on the stacked head; zero unresolved threads. | `blocked`; parent #109 is mutable and required evidence is `absent`. | + +## Open issue continuation snapshot + +| Issue | Status-qualified continuation | +|---:|---| +| Issue #86 | `active_pr` in #90; dynamic NIM benchmark claims remain unshipped. | +| Issue #95 | `active_pr` in #96; close only after protected integration and acceptance. | +| Issue #102 | `planned`; equivalent-endpoint racing must wait for the accepted security and review boundary. | +| Issue #103 | `planned`; release readiness must fail closed on exact-head checks and qualifying reviews. | + +## Operational and release acceptance + +The protected-main revision remained unchanged from the preceding acceptance +run: 300 functional tests passed, while owned production coverage was 88% and +public-docstring coverage was 95.4%. Those values are protected-main facts, +not failures attributed to an active branch. They remain below the accepted +100% production statement/branch and public-docstring release contract. + +No protected merge, release tag, package publication, SBOM/provenance release +receipt, reproducibility receipt, migration/rollback exercise, certification, +or production SLO acceptance was observed. Successful active-PR checks cannot +fill those release-evidence gaps. + +## Next acceptance boundary + +Preserve stable heads while the external review/governance path is pending. +The next merge decision must refetch the exact contributor head, exact base +branch tip, protected target tip, required contexts and their checked-out +commits, formal reviews, unresolved threads, rulesets, and release evidence. +After #96 reaches protected `main`, rebuild each dependent PR in dependency +order and regenerate every check and review; no row in this appendix transfers. diff --git a/docs/evidence/README.md b/docs/evidence/README.md new file mode 100644 index 000000000..df12a4797 --- /dev/null +++ b/docs/evidence/README.md @@ -0,0 +1,14 @@ +# Dated evidence appendices + +This directory is the only documentation authority for volatile commit SHAs, +workflow and job IDs, dated pull-request inventories, and central dependency +snapshots. Appendices are immutable historical observations, not current merge +authority. Every readiness decision must refetch the live branch, checks, +reviews, threads, and protected base. + +| Appendix | Scope | Authority | +|---|---|---| +| [2026-08-12 continuation audit](2026-08-12-continuation-audit.md) | Latest collected evidence: protected refs, open-PR and issue continuation, central prerequisites, and release acceptance | Historical upon any relevant state change | +| [2026-08-11 documentation audit](2026-08-11-documentation-audit.md) | Pre-repair sufficiency assessment, open-PR inventory, and central prerequisite snapshot | Historical evidence only | + +Durable requirement relationships remain in [Traceability](../TRACEABILITY.md). diff --git a/docs/fuzzing.md b/docs/fuzzing.md index 9897b2bd2..e2b74bfa8 100644 --- a/docs/fuzzing.md +++ b/docs/fuzzing.md @@ -63,5 +63,5 @@ as artifacts. ## Background For the theory behind coverage-guided greybox fuzzing, see -[`papers/fuzzing-art-science-engineering-manes-2019.pdf`](papers/fuzzing-art-science-engineering-manes-2019.pdf) +[Manès et al., *The Art, Science, and Engineering of Fuzzing*](https://doi.org/10.1109/TSE.2019.2946563) (Manès et al., *The Art, Science, and Engineering of Fuzzing: A Survey*). diff --git a/docs/i18n_design.md b/docs/i18n_design.md index b61208c78..5e0902d50 100644 --- a/docs/i18n_design.md +++ b/docs/i18n_design.md @@ -1,5 +1,8 @@ # i18n Design +**Document state:** `implemented_on_protected_main` for the inline locale +bundles and `planned` for the adoption candidate described below. + ## Locales - `en`: default and fallback locale. @@ -10,8 +13,8 @@ UI messages are locale bundles: - REST: `GET /api/v1/locale_bundles/{locale_code}` -- Admin runtime: inlined bundle for the dependency-free prototype. -- Production runtime: i18next resources loaded over HTTP. +- Admin runtime: inlined bundle in the standalone runtime. +- Planned web client: i18next resources loaded over HTTP. ## Key Rules @@ -25,7 +28,8 @@ UI messages are locale bundles: `contextual_orchestrator.admin.ADMIN_TRANSLATIONS` contains English and Korean bundles. The admin console switches language without a page reload. -## Production Library Target - -Use i18next for resource loading, fallback language, interpolation, language detection, and runtime switching. React-admin should receive an `i18nProvider` backed by the same bundles. +## Planned adoption candidate +i18next and React-admin are planned adoption candidates. Adopt them for a +separately built web client only with migration, rollback, and parity evidence; +they do not own the current inline admin call path. diff --git a/docs/library_research.md b/docs/library_research.md index 42c7fa95c..ef22d6e14 100644 --- a/docs/library_research.md +++ b/docs/library_research.md @@ -1,25 +1,34 @@ -# Library Research +# Library Research and Adoption Status -The design researched existing libraries before adding code. The repository keeps the runtime dependency-free for the current lab, but the enterprise implementation target is explicit. +This record compares library candidates with the code that is actually shipped +on protected `main`. Status labels use the canonical vocabulary: an optional +dependency is not an implemented integration, and a design choice is not +presented as production behavior merely because its package is installable. ## Selected Stack -| Area | Library | Decision | Evidence | +| Area | Status | Current decision | Evidence | |---|---|---|---| -| REST API | [FastAPI](https://github.com/fastapi/fastapi) | Use when the API moves beyond the current stdlib prototype. | FastAPI provides request validation with Pydantic models, standard status/response declarations, and OpenAPI/JSON Schema generation. Context7: `/fastapi/fastapi`. | -| Admin console | [React-admin](https://github.com/marmelab/react-admin) | Use for production CRUD/admin surfaces. | React-admin has `Admin`, `Resource`, `dataProvider`, `authProvider`, `i18nProvider`, dashboard, layout, and custom route hooks. Context7: `/marmelab/react-admin`. | -| i18n | [i18next](https://github.com/i18next/i18next) | Use for shared web translation runtime, especially outside React-admin defaults. | i18next supports resource bundles, `fallbackLng`, interpolation, language detection, and runtime `changeLanguage`. Context7: `/i18next/i18next`. | -| Persistence | [SQLAlchemy 2.x](https://docs.sqlalchemy.org/orm/) | Use for Python domain persistence. | Official docs cover ORM mapped classes and sessions. | -| Migrations | [Alembic](https://alembic.sqlalchemy.org/) | Use for schema migration lifecycle. | Alembic is the SQLAlchemy migration tool and supports autogenerated migrations from metadata. | -| Database | [PostgreSQL](https://www.postgresql.org/docs/current/sql-syntax-lexical.html) | Default relational store. | PostgreSQL identifiers allow letters, digits, and underscores; the project standardizes on unquoted lower snake_case. | -| API contract | [OpenAPI 3.1](https://spec.openapis.org/oas/v3.1.0.html) | Contract format for API review and client generation. | OAS defines a language-agnostic HTTP API description for humans and machines. | +| HTTP runtime | `implemented_on_protected_main` | The production path uses `ThreadingHTTPServer` with bounded request handling. | `contextual_orchestrator/server.py` constructs the server and handler directly. | +| API framework | `planned` | FastAPI and Uvicorn remain candidates. The optional `api` extra makes them installable, but production request dispatch does not dispatch through FastAPI. | `pyproject.toml` declares the extra; production modules do not import FastAPI or Uvicorn. | +| API contract | `implemented_on_protected_main` | Keep the handwritten OpenAPI 3.1 contract synchronized with runtime dispatch. | `contextual_orchestrator/api_contract.py` and contract tests verify the current routes. | +| Admin console | `implemented_on_protected_main` / `planned` | The static admin UI is current. React-admin remains a planned replacement only if CRUD and external identity integration justify it. | `contextual_orchestrator/admin.py`; React-admin is not a runtime dependency. | +| Internationalization | `implemented_on_protected_main` / `planned` | Current English/Korean resources are inlined. i18next remains planned for a separately built web client. | Admin locale tests cover the current bundle; i18next is not installed. | +| Persistence | `implemented_on_protected_main` / `accepted_architecture` | Current code supports SQLite state, a PEP-249 ledger, and optional direct `psycopg` credential storage. SQLAlchemy ORM and Alembic remain accepted architecture, not current runtime behavior. | The optional `db` extra installs SQLAlchemy, Alembic, and psycopg, but production code does not use SQLAlchemy ORM or Alembic migrations. | +| Database model | `implemented_on_protected_main` / `accepted_architecture` | Runtime adapters use SQLite or PostgreSQL-compatible paths; the normalized model in `docs/database_design.sql` remains an accepted migration target. | `docs/ERD.md` distinguishes runtime, external, conceptual, and planned entities. | -## Ponytail Decision +## Dependency-Adoption Rule No new dependency is added until it carries real product weight: -- Current prototype: stdlib server, handwritten OpenAPI, static admin UI. -- First enterprise cut: FastAPI + React-admin + i18next + PostgreSQL + SQLAlchemy + Alembic. +- Current implementation: stdlib `ThreadingHTTPServer`, handwritten OpenAPI, + static admin UI, inlined locales, SQLite/PEP-249 state paths, and an optional + direct-psycopg credential backend. +- Optional extras advertise installable compatibility surfaces. They do not + prove that FastAPI, SQLAlchemy ORM, or Alembic owns a production call path. +- Adopt FastAPI, React-admin, i18next, SQLAlchemy, or Alembic only with a bounded + product requirement, migration and rollback evidence, and tests proving the + new authority boundary. - Do not add provider SDKs until raw OpenAI-compatible HTTP is insufficient. Skipped: custom admin framework, custom i18n engine, custom migration engine. @@ -50,8 +59,8 @@ Extraction triggers: - Security review requires a reusable, locked core package with independent provenance. -Until those triggers exist, Ponytail recommends strengthening the current -single-repo product instead of splitting it. +Until those triggers exist, the accepted architecture is to strengthen the +current single-repository product instead of splitting it. ## Required For New Designs diff --git a/docs/papers/README.md b/docs/papers/README.md index 65a89d2af..fb3885a31 100644 --- a/docs/papers/README.md +++ b/docs/papers/README.md @@ -1,48 +1,17 @@ -# Papers grounding the cost-review + routing hub - -These papers ground the design of the LLM **cost review** ledger and the -**sync-vs-batch / upstream** routing added in `feat/cost-review-and-batch-routing`. -All three are arXiv preprints distributed under licenses that permit -redistribution; each is cited below with its arXiv identifier. - -## Cost optimisation - -- **FrugalGPT: How to Use Large Language Models While Reducing Cost and - Improving Performance** — Lingjiao Chen, Matei Zaharia, James Zou. arXiv:2305.05176, 2023. - `frugalgpt-cost-2305.05176.pdf` - Motivates the **configurable price table + per-request cost accounting** and - cost-optimising model selection: cost varies by orders of magnitude across - providers/models, so a gateway should price each request and route to the - cheapest capable upstream. Distributed under arXiv's non-exclusive license to - distribute (arXiv perpetual, non-exclusive license 1.0). - -## Query routing (which upstream / which tier) - -- **RouteLLM: Learning to Route LLMs with Preference Data** — Isaac Ong, Amjad - Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. - Waleed Kadous, Ion Stoica. arXiv:2406.18665, 2024. - `routellm-routing-2406.18665.pdf` - Grounds the **routing decision** layer (`RoutingPolicy` + cost-aware upstream - selection): route strong/weak model choices to hit a cost/quality target. - arXiv preprint; distributed under the arXiv non-exclusive distribution license. - -- **Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing** — Dujian Ding, - Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, - Laks V. S. Lakshmanan, Ahmed Hassan Awadallah. arXiv:2404.14618 (ICLR 2024). - `hybrid-llm-query-routing-2404.14618.pdf` - Grounds **latency-tolerant vs interactive routing** and the sync/batch split: - route easy/bulk queries to the cheaper path, keep hard/interactive queries on - the responsive path. Distributed under the arXiv non-exclusive license / - CC BY as marked on arXiv. - -## Batch execution / load balancing - -The external `pg-llm-batch` service carries its own grounding papers, including -PagedAttention / vLLM (2309.06180) and DeepSpeed-FastGen (2401.08671), which -motivate throughput-oriented **batched** inference and the load-balancing that -makes the latency-tolerant batch route economical. Those sources are referenced -but not vendored here so this repository remains one deployable control plane. - -> Citations are provided for scholarly attribution. Redistribution here relies -> on the arXiv non-exclusive distribution license each author granted; no -> GPL/AGPL-licensed material is vendored anywhere in this repository. +# Paper source and license index + +The papers below motivate routing, evaluation, and fuzzing hypotheses. This +repository links to authoritative sources instead of vendoring article PDFs +unless a separate redistribution permission or legal review is recorded. + +| Work | Product use | Authoritative source | License note | +|---|---|---|---| +| FrugalGPT (Chen et al., 2023) | Cost/quality frontier and cascade evaluation. | https://doi.org/10.48550/arXiv.2305.05176 | The arXiv non-exclusive grant is to arXiv; it does not itself grant downstream repository redistribution. | +| RouteLLM (Ong et al., 2024) | Strong/weak model routing and preference-based evaluation. | https://doi.org/10.48550/arXiv.2406.18665 | The arXiv non-exclusive grant is not downstream permission. | +| Hybrid LLM (Ding et al., 2024) | Difficulty-aware small/large-model routing. | Paper: https://openreview.net/forum?id=02f3mUtqnM; arXiv source/license page: https://arxiv.org/abs/2404.14618 | The arXiv copy is marked CC BY-NC-ND 4.0; commercial repository redistribution is not assumed. | +| The Art, Science, and Engineering of Fuzzing (Manès et al., 2019) | Fuzz target selection, oracles, corpora, and lifecycle. | https://doi.org/10.1109/TSE.2019.2946563 | Publisher/authors retain rights unless separate permission applies. | + +Hybrid LLM does not establish an interactive-versus-batch split. That split is +a repository product inference and needs its own tests and operational evidence. +Full APA 7 references and source-license authorities are in +[the reference index](../REFERENCES.md). diff --git a/docs/papers/frugalgpt-cost-2305.05176.pdf b/docs/papers/frugalgpt-cost-2305.05176.pdf deleted file mode 100644 index fd20d3e8d..000000000 Binary files a/docs/papers/frugalgpt-cost-2305.05176.pdf and /dev/null differ diff --git a/docs/papers/fuzzing-art-science-engineering-manes-2019.pdf b/docs/papers/fuzzing-art-science-engineering-manes-2019.pdf deleted file mode 100644 index b0cd20886..000000000 Binary files a/docs/papers/fuzzing-art-science-engineering-manes-2019.pdf and /dev/null differ diff --git a/docs/papers/hybrid-llm-query-routing-2404.14618.pdf b/docs/papers/hybrid-llm-query-routing-2404.14618.pdf deleted file mode 100644 index 8824626be..000000000 Binary files a/docs/papers/hybrid-llm-query-routing-2404.14618.pdf and /dev/null differ diff --git a/docs/papers/routellm-routing-2406.18665.pdf b/docs/papers/routellm-routing-2406.18665.pdf deleted file mode 100644 index d78738fb1..000000000 Binary files a/docs/papers/routellm-routing-2406.18665.pdf and /dev/null differ diff --git a/docs/product_planning.md b/docs/product_planning.md index 74a4aece7..98233f820 100644 --- a/docs/product_planning.md +++ b/docs/product_planning.md @@ -56,7 +56,7 @@ Enterprise teams want the benefit of collective model intelligence without makin - No learned coordinator training. Keep deterministic routing until there is an evaluation set proving it is the bottleneck. - No visual workflow builder. Tables and trace details are enough until operators need to author complex topologies. - No recursive topology UI. Conductor recursion is a future scaling knob, not an MVP control. -- No billing, SSO, or RBAC implementation in the stdlib lab. Document the need; add it with the enterprise stack. +- No billing or SSO implementation. The standalone runtime exposes coarse admin and inference bearer scopes, but no tenant-aware RBAC; the host owns enterprise identity and tenancy. ## Acceptance Criteria diff --git a/docs/rest_api_design.md b/docs/rest_api_design.md index 9378e5a37..a5140c3fb 100644 --- a/docs/rest_api_design.md +++ b/docs/rest_api_design.md @@ -1,5 +1,8 @@ # REST API Design +**Document state:** `implemented_on_protected_main` for current endpoints and +`planned` for the adoption candidate described below. + ## Rules - API version prefix: `/api/v1`. @@ -58,7 +61,7 @@ ## Product Planning Additions (Implemented) -These product surfaces are now implemented in this prototype: +These product surfaces are implemented in the standalone runtime: | Method | Path | Purpose | Paper Basis | |---|---|---|---| @@ -93,6 +96,9 @@ These product surfaces are now implemented in this prototype: | `GET` | `/api/v1/commercial_due_diligence_rooms/latest` | Produce the buyer due diligence room that ties purchase approval, runtime API evidence, admin trace/access evidence, security, commercial terms, value analytics, implementation readiness, Figma, review-process policy, packaging decision, and buyer/external missing artifacts into one runtime diligence artifact. | Fugu API adoption; TRINITY verification; Conductor trace/access evidence; buyer diligence committee review. | | `GET` | `/api/v1/commercial_investment_committee_memos/latest` | Produce the investment committee memo that ties due diligence, purchase approval, financial case, risk/security, commercial terms, implementation readiness, Figma, review-process policy, packaging decision, and buyer/external approval conditions into one executive recommendation artifact. | Fugu API adoption; TRINITY verification; Conductor trace/access evidence; executive investment committee review. | -## Production Library Target +## Planned adoption candidate -FastAPI should replace the current stdlib HTTP adapter when the API needs authentication, richer OpenAPI schema generation, dependency injection, and typed request/response models. +FastAPI is one of the optional compatibility extras. Adopt it only when the API +needs richer OpenAPI generation, dependency injection, and typed request and +response models, with migration, rollback, and contract evidence. It does not +own the current HTTP call path. diff --git a/tests/test_documentation_contract.py b/tests/test_documentation_contract.py new file mode 100644 index 000000000..14b14bcc1 --- /dev/null +++ b/tests/test_documentation_contract.py @@ -0,0 +1,1022 @@ +"""Machine-check the canonical product and architecture documentation graph.""" + +import ast +import re +from pathlib import Path + +import pytest + + +ROOT_DIR = Path(__file__).resolve().parents[1] + +STATUS_VOCABULARY = { + "implemented_on_protected_main", + "active_pr", + "accepted_architecture", + "planned", + "research_only", + "superseded", + "out_of_scope", +} + +ADR_FILES = [ + "0001-route-conduct-test-time-compute.md", + "0002-provider-neutral-transport-trust.md", + "0003-workflow-access-and-reasoning-control.md", + "0004-kv-credential-bootstrap.md", + "0005-sync-batch-pg-llm-batch.md", + "0006-honest-cost-and-benchmark-evidence.md", + "0007-free-first-fallback.md", + "0008-state-persistence-and-retention.md", + "0009-purpose-bound-pii-protection.md", + "0010-independent-review-and-evidence.md", + "0011-release-coverage-and-provenance.md", + "0012-standalone-and-cwl-boundary.md", + "0013-database-naming-and-migration.md", + "0014-scientific-computation-ownership.md", + "0015-provider-egress-response-trust.md", + "0016-complete-coverage-docstrings.md", +] + +REQUIRED_FILES = [ + "ARCHITECTURE.md", + "docs/README.md", + "docs/PRD.md", + "docs/TRD.md", + "docs/UML.md", + "docs/ERD.md", + "docs/TRACEABILITY.md", + "docs/evidence/README.md", + "docs/evidence/2026-08-11-documentation-audit.md", + "docs/evidence/2026-08-12-continuation-audit.md", + "docs/THREAT_MODEL.md", + "docs/TEST_STRATEGY.md", + "docs/OPERABILITY.md", + "docs/INCIDENT_RUNBOOK.md", + "docs/RELEASE_GUIDE.md", + "docs/REFERENCES.md", + "docs/adr/README.md", + "SECURITY.md", +] + [f"docs/adr/{filename}" for filename in ADR_FILES] + +ADR_HEADINGS = [ + "## Status", + "## Context and decision drivers", + "## Considered alternatives", + "## Decision", + "## Consequences", + "## Failure and recovery", + "## Security, privacy, and governance impact", + "## Compatibility and migration", + "## Verification and acceptance", + "## Rollback and supersession", + "## References", +] + +CANONICAL_FILES = [ + "ARCHITECTURE.md", + "docs/README.md", + "docs/PRD.md", + "docs/TRD.md", + "docs/UML.md", + "docs/ERD.md", + "docs/TRACEABILITY.md", + "docs/THREAT_MODEL.md", + "docs/TEST_STRATEGY.md", + "docs/OPERABILITY.md", + "docs/INCIDENT_RUNBOOK.md", + "docs/RELEASE_GUIDE.md", + "docs/REFERENCES.md", + "docs/adr/README.md", +] + [f"docs/adr/{filename}" for filename in ADR_FILES] + +LINK_CHECK_FILES = CANONICAL_FILES + [ + "AGENTS.md", + "CLAUDE.md", + "README.md", + "SECURITY.md", + "docs/evidence/README.md", + "docs/evidence/2026-08-11-documentation-audit.md", + "docs/evidence/2026-08-12-continuation-audit.md", + "docs/architecture.md", + "docs/fuzzing.md", + "docs/papers/README.md", +] + +AUDITED_OPEN_PR_NUMBERS = { + 63, + 66, + 69, + 71, + 75, + 82, + 83, + 90, + 94, + 96, + 99, + 104, + 105, + 107, + 108, + 109, + 110, +} + + +def read_text(relative_path: str) -> str: + """Return one repository file as UTF-8 text.""" + + return (ROOT_DIR / relative_path).read_text(encoding="utf-8") + + +def canonical_text() -> str: + """Return durable canonical documents without the dated evidence appendix.""" + + return "\n".join(read_text(path) for path in CANONICAL_FILES) + + +DATED_EVIDENCE_APPENDIX = "docs/evidence/2026-08-11-documentation-audit.md" +LATEST_EVIDENCE_APPENDIX = "docs/evidence/2026-08-12-continuation-audit.md" + + +def class_method_names(relative_path: str, class_name: str) -> set[str]: + """Return methods declared directly on one runtime class.""" + + tree = ast.parse(read_text(relative_path)) + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == class_name: + return { + child.name + for child in node.body + if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + raise AssertionError(f"{class_name} is absent from {relative_path}") + + +def validate_mermaid_subset(block: str, source_path: str) -> None: + """Parse the Mermaid subset used by the canonical architecture documents.""" + + lines = [line.strip() for line in block.splitlines() if line.strip()] + assert lines, f"{source_path} contains an empty Mermaid block" + diagram_type, body = lines[0], lines[1:] + assert body, f"{source_path} contains an empty {diagram_type} diagram" + for line in body: + assert line.count('"') % 2 == 0, f"{source_path}: unbalanced quote in {line!r}" + + if diagram_type.startswith("flowchart "): + assert re.fullmatch(r"flowchart (?:TB|TD|BT|LR|RL)", diagram_type) + node = r"[A-Za-z_][A-Za-z0-9_]*(?:\[[^\]]+\])?" + edge = re.compile( + rf"{node}\s+(?:-->(?:\|[^|]+\|)?|-\..+\.->)\s*{node}" + ) + subgraph_depth = 0 + for line in body: + if line.startswith("subgraph "): + assert re.fullmatch(rf"subgraph {node}", line), ( + f"{source_path}: unsupported subgraph syntax {line!r}" + ) + subgraph_depth += 1 + elif line == "end": + assert subgraph_depth > 0, f"{source_path}: unmatched flowchart end" + subgraph_depth -= 1 + elif re.fullmatch(edge, line): + continue + else: + assert re.fullmatch(node, line), ( + f"{source_path}: unsupported flowchart statement {line!r}" + ) + assert subgraph_depth == 0, f"{source_path}: unclosed flowchart subgraph" + return + + if diagram_type == "sequenceDiagram": + participants: set[str] = set() + control_stack: list[str] = [] + for line in body: + declaration = re.fullmatch( + r"(?:actor|participant)\s+([A-Za-z_][A-Za-z0-9_]*)(?:\s+as\s+.+)?", + line, + ) + if declaration: + participants.add(declaration.group(1)) + continue + control = re.match(r"(alt|opt|loop|par|critical|break|rect)\s+.+", line) + if control: + control_stack.append(control.group(1)) + continue + if line.startswith("else "): + assert control_stack and control_stack[-1] == "alt", ( + f"{source_path}: else outside alt" + ) + continue + if line == "end": + assert control_stack, f"{source_path}: unmatched sequence end" + control_stack.pop() + continue + message = re.fullmatch( + r"([A-Za-z_][A-Za-z0-9_]*)\s*(?:->>|-->>)\s*" + r"([A-Za-z_][A-Za-z0-9_]*)\s*:\s*.+", + line, + ) + assert message, f"{source_path}: unsupported sequence statement {line!r}" + sender, receiver = message.groups() + assert sender in participants, f"{source_path}: undeclared participant {sender}" + assert receiver in participants, f"{source_path}: undeclared participant {receiver}" + assert not control_stack, f"{source_path}: unclosed sequence control block" + return + + if diagram_type == "stateDiagram-v2": + transition = re.compile( + r"(?:\[\*\]|[A-Za-z_][A-Za-z0-9_]*)\s+-->\s+" + r"(?:\[\*\]|[A-Za-z_][A-Za-z0-9_]*)(?::\s+.+)?" + ) + for line in body: + assert re.fullmatch(transition, line), ( + f"{source_path}: unsupported state statement {line!r}" + ) + return + + if diagram_type == "erDiagram": + in_entity = False + for line in body: + if re.fullmatch(r"[A-Z][A-Z0-9_]* \{", line): + assert not in_entity, f"{source_path}: nested ER entity" + in_entity = True + elif line == "}": + assert in_entity, f"{source_path}: unmatched ER entity close" + in_entity = False + elif in_entity: + assert re.fullmatch( + r"[A-Za-z_][A-Za-z0-9_]*(?:\([^)]*\))?\s+" + r"[A-Za-z_][A-Za-z0-9_]*(?:\s+.+)?", + line, + ), f"{source_path}: invalid ER attribute {line!r}" + else: + assert re.fullmatch( + r"[A-Z][A-Z0-9_]*\s+[|o{}]+--[|o{}]+\s+" + r"[A-Z][A-Z0-9_]*\s*:\s*.+", + line, + ), f"{source_path}: invalid ER relationship {line!r}" + assert not in_entity, f"{source_path}: unclosed ER entity" + return + + raise AssertionError(f"{source_path}: unsupported Mermaid type {diagram_type!r}") + + +def test_required_canonical_files_are_present_and_indexed() -> None: + """Require one discoverable authority for every requested document family.""" + + missing = [path for path in REQUIRED_FILES if not (ROOT_DIR / path).is_file()] + assert missing == [] + + index_text = read_text("docs/README.md") + for link in ( + "[PRD](PRD.md)", + "[TRD](TRD.md)", + "[Architecture](../ARCHITECTURE.md)", + "[UML](UML.md)", + "[ERD](ERD.md)", + "[ADR index](adr/README.md)", + "[Traceability](TRACEABILITY.md)", + "[Dated evidence appendices](evidence/README.md)", + "[Threat model](THREAT_MODEL.md)", + "[Test strategy](TEST_STRATEGY.md)", + "[Operability](OPERABILITY.md)", + "[Incident runbook](INCIDENT_RUNBOOK.md)", + "[Release guide](RELEASE_GUIDE.md)", + "[References](REFERENCES.md)", + ): + assert index_text.count(link) == 1 + assert "[docs/README.md](docs/README.md)" in read_text("README.md") + + +def test_release_guide_binds_source_artifact_migration_and_operations() -> None: + """Require an executable release handoff without claiming branch evidence shipped.""" + + release_text = read_text("docs/RELEASE_GUIDE.md") + for heading in ( + "## Authority and release identity", + "## Admission checklist", + "## Build and provenance procedure", + "## Migration and rollback procedure", + "## Publication and deployment procedure", + "## Protected-main operational acceptance", + "## Abort and recovery conditions", + ): + assert heading in release_text + for required_term in ( + "protected `main`", + "exact source commit", + "CycloneDX SBOM", + "provenance", + "reproducible", + "independent non-author approval", + "expand/backfill/contract", + "rollback", + "artifact digest", + "protected-main", + "does not claim", + ): + assert required_term in release_text + + +def test_status_vocabulary_product_scope_and_prompt_continuity_are_explicit() -> None: + """Keep shipped/proposed work distinct and prevent audit-only early stops.""" + + index_text = read_text("docs/README.md") + for status in STATUS_VOCABULARY: + assert f"`{status}`" in index_text + + product_text = read_text("docs/PRD.md") + for requirement_id in range(1, 11): + assert f"PRD-{requirement_id:03d}" in product_text + for status in ( + "implemented_on_protected_main", + "active_pr", + "planned", + "out_of_scope", + ): + assert f"`{status}`" in product_text + + prompt_text = read_text("AGENTS.md") + assert "## Execution continuity" in prompt_text + assert "intermediate work" in prompt_text + + +def test_active_local_mlx_work_is_status_qualified_across_canonical_docs() -> None: + """Keep the active local-provider slice distinct from shipped behavior.""" + + canonical_paths = ( + "ARCHITECTURE.md", + "docs/PRD.md", + "docs/TRD.md", + "docs/TRACEABILITY.md", + ) + for path in canonical_paths: + matching_lines = [ + line + for line in read_text(path).splitlines() + if "PR #109" in line and "MLX" in line + ] + assert matching_lines, f"{path} must trace the PR #109 MLX slice" + assert any("`active_pr`" in line for line in matching_lines), ( + f"{path} must label the PR #109 MLX slice active_pr" + ) + + trd_lines = [ + line + for line in read_text("docs/TRD.md").splitlines() + if "PR #109" in line and "MLX" in line + ] + assert any("FR-015" in line for line in trd_lines) + + +def test_root_readme_uses_current_buyer_facing_product_identity() -> None: + """Keep the entry point aligned with the governed orchestration product.""" + + readme = read_text("README.md") + normalized_readme = " ".join(readme.split()) + assert ( + "Provider-neutral OpenAI-compatible orchestration control plane that " + "routes, conducts, verifies, and synthesizes work across governed model " + "agents." + in normalized_readme + ) + assert "Stdlib Python lab for a single API" not in readme + assert "hardened for local deployment" in readme + assert "hardened for local lab use" not in readme + assert "stdlib lab" not in readme.lower() + assert "standalone deployment" in readme.lower() + assert "This is not a Sakana AI product" not in readme + assert ( + "independently implemented from published orchestration concepts" + in normalized_readme + ) + assert ( + "no third-party trained model weights or proprietary artifacts" + in normalized_readme + ) + + +def test_product_planning_qualifies_enterprise_identity_and_authorization() -> None: + """Prevent supporting product plans from overstating the auth boundary.""" + + planning = read_text("docs/product_planning.md").lower() + + assert "stdlib lab" not in planning + assert "coarse admin and inference bearer scopes" in planning + assert "no tenant-aware rbac" in planning + assert "host owns enterprise identity and tenancy" in planning + + +def test_library_research_uses_current_stack_and_status_vocabulary() -> None: + """Supporting library research must describe the live stack without lab names.""" + + research = read_text("docs/library_research.md") + + for stale_term in ("current lab", "current stdlib prototype", "Ponytail"): + assert stale_term not in research + + for required_term in ( + "implemented_on_protected_main", + "accepted_architecture", + "planned", + "ThreadingHTTPServer", + "handwritten OpenAPI", + "static admin UI", + "optional `api` extra", + "does not dispatch through FastAPI", + "optional `db` extra", + "does not use SQLAlchemy ORM or Alembic migrations", + ): + assert required_term in research + + +def test_agent_and_cdd_guidance_use_current_product_and_adoption_language() -> None: + """Keep agent and CDD guidance aligned with the current product authority.""" + + claude = read_text("CLAUDE.md") + workflow = read_text("conductor/workflow.md") + stack = read_text("conductor/tech-stack.md") + combined = "\n".join((claude, workflow, stack)) + normalized = " ".join(combined.split()).lower() + + for stale_term in ( + "stdlib-Python lab", + "Ponytail design gate", + "Ponytail Design Gate", + "after this lab hardens", + ): + assert stale_term not in combined + + for required_term in ( + "provider-neutral OpenAI-compatible orchestration control plane", + "dependency-adoption gate", + "current implementation dependencies", + "planned adoption candidates", + "optional extras are installable compatibility surfaces", + ): + assert required_term.lower() in normalized + + +def test_supporting_runtime_docs_use_status_qualified_product_language() -> None: + """Keep supporting runtime guides explicit about shipped and planned scope.""" + + analytics = read_text("docs/analytics_spec.md") + api_design = read_text("docs/rest_api_design.md") + i18n = read_text("docs/i18n_design.md") + combined = "\n".join((analytics, api_design, i18n)) + normalized = " ".join(combined.split()).lower() + + for stale_term in ( + "prototype", + "Production Library Target", + "dependency-free", + ): + assert stale_term not in combined + + assert combined.count("**Document state:**") == 3 + for required_term in ( + "implemented_on_protected_main", + "standalone runtime", + "not production telemetry", + "planned adoption candidate", + "optional compatibility extras", + ): + assert required_term.lower() in normalized + + +def test_adr_index_and_schema_are_consistent() -> None: + """Require every decision to be uniquely indexed, status-bearing, and recoverable.""" + + index_text = read_text("docs/adr/README.md") + numbers = [] + for filename in ADR_FILES: + number = filename.split("-", 1)[0] + numbers.append(number) + assert index_text.count(f"({filename})") == 1 + body = read_text(f"docs/adr/{filename}") + for heading in ADR_HEADINGS: + assert heading in body, f"{filename} lacks {heading}" + status_section = body.split("## Status", 1)[1].split("\n## ", 1)[0] + assert any(f"`{status}`" in status_section for status in STATUS_VOCABULARY) + assert len(numbers) == len(set(numbers)) + + +def test_mermaid_blocks_parse_supported_syntax_and_cover_required_views() -> None: + """Parse the supported Mermaid subset and require all architecture views.""" + + diagram_sources = { + "ARCHITECTURE.md": "flowchart", + "docs/UML.md": "sequenceDiagram", + "docs/ERD.md": "erDiagram", + "docs/TRACEABILITY.md": "flowchart", + } + all_diagrams = [] + for path, required_type in diagram_sources.items(): + source = read_text(path) + blocks = re.findall(r"```mermaid\s*\n(.*?)```", source, flags=re.DOTALL) + assert len(blocks) == source.count("```mermaid"), path + assert any(block.lstrip().startswith(required_type) for block in blocks), path + for block in blocks: + validate_mermaid_subset(block, path) + all_diagrams.extend(blocks) + assert any(block.lstrip().startswith("stateDiagram-v2") for block in all_diagrams) + + invalid_sequence = """sequenceDiagram + actor Caller + Caller->>Undeclared: request + """ + try: + validate_mermaid_subset(invalid_sequence, "negative fixture") + except AssertionError: + pass + else: # pragma: no cover - proves the validator is fail-closed + raise AssertionError("Mermaid validator accepted an undeclared participant") + + +def test_live_names_routes_and_physical_data_objects_are_not_stale() -> None: + """Tie architecture names and ERD claims to protected-main source strings.""" + + durable_text = canonical_text() + assert "TaskOrchestrator" in durable_text + assert "CostRoutingCoordinator" in durable_text + assert "contextual_orchestrator.orchestrator.Agent" not in durable_text + assert "contextual_orchestrator.orchestrator.Orchestrator" not in durable_text + assert "Orchestrator.route_once" not in durable_text + + uml_text = read_text("docs/UML.md") + coordinator_methods = class_method_names( + "contextual_orchestrator/cost_router.py", "CostRoutingCoordinator" + ) + for method_name in ("complete", "poll_batch", "retrieve_batch"): + assert method_name in coordinator_methods + assert f"{method_name}(...)" in uml_text + assert "route_request(...)" not in uml_text + assert "Server->>Router: poll_batch(...) or retrieve_batch(...)" in uml_text + assert "Router->>Backend: poll(...) or retrieve(...)" in uml_text + + server_source = read_text("contextual_orchestrator/server.py") + trd_text = read_text("docs/TRD.md") + for endpoint in ( + "/healthz", + "/v1/chat/completions", + "/v1/responses", + "/v1/batch/embeddings", + "/api/v1/batch_routing_jobs", + "/api/v1/workflow_runs", + ): + assert endpoint in server_source + assert endpoint in trd_text + + physical_sources = "\n".join( + read_text(path) + for path in ( + "contextual_orchestrator/orchestrator.py", + "contextual_orchestrator/credentials.py", + "contextual_orchestrator/cost_ledger.py", + ) + ) + erd_text = read_text("docs/ERD.md") + for object_name in ( + "agent_pool", + "records", + "provider_credentials", + "cost_attribution_dimensions", + "llm_price_entries", + "llm_usage_records", + ): + assert f"CREATE TABLE IF NOT EXISTS {object_name}" in physical_sources + assert f"`{object_name}`" in erd_text + for classification in ( + "persisted_runtime", + "in_memory", + "external_owned", + "accepted_target", + "active_pr", + ): + assert f"`{classification}`" in erd_text + + +def test_current_evidence_gaps_are_not_promoted_into_capabilities() -> None: + """Keep cost, stream, batch, OpenAPI, and provider-selection gaps explicit.""" + + text = canonical_text() + for term in ( + "two unsynchronized cost authorities", + "missing price as zero", + "sql price table is dormant", + "passthrough", + "route streaming", + "process-local", + "openapi", + "cheapest_upstream()", + "not learned, price-aware, or load-balanced", + ): + assert term in text.lower() + + architecture_text = " ".join(read_text("ARCHITECTURE.md").lower().split()) + threat_text = " ".join(read_text("docs/THREAT_MODEL.md").lower().split()) + readme_text = " ".join(read_text("README.md").lower().split()) + assert "no dedicated trace scope" in architecture_text + assert "inference-scoped caller may request" in architecture_text + assert "no enforced provider-response byte cap" in threat_text + assert "default in-memory credential backend is process-local" in readme_text + assert "route-stream workflow runs remain memory-only" in readme_text + assert "budget precheck is process-local and non-atomic" in readme_text + assert "ordinary non-stream orchestrated calls" in readme_text + + sync_batch_adr = " ".join( + read_text("docs/adr/0005-sync-batch-pg-llm-batch.md").lower().split() + ) + assert "ordinary coordinator sync" in sync_batch_adr + assert "passthrough and route streaming bypass" in sync_batch_adr + route_conduct_adr = " ".join( + read_text("docs/adr/0001-route-conduct-test-time-compute.md").lower().split() + ) + assert "snapshotted policy" in route_conduct_adr + assert "a versioned policy" not in route_conduct_adr + + +def test_database_naming_exception_is_bounded() -> None: + """Do not hide the one-word legacy table or normalize future ambiguity.""" + + erd_text = read_text("docs/ERD.md") + adr_text = read_text("docs/adr/0013-database-naming-and-migration.md") + for text in (erd_text, adr_text): + assert "two-or-more-word snake_case" in text + assert "`records`" in text + assert "technical debt" in text + assert "runtime_records" in text + + +def test_local_markdown_links_resolve() -> None: + """Reject broken relative links in the canonical graph.""" + + link_pattern = re.compile(r"(? None: + """Keep research, licensing, and revision evidence under distinct authority.""" + + references = read_text("docs/REFERENCES.md") + for term in ( + "Fugu", + "Conductor", + "TRINITY", + "RFC 8259", + "NIST SP 800-218A", + "42001:2023", + "OpenAPI Specification", + "3.1.0", + "CSAP", + "non-exclusive grant to arXiv", + ): + assert term in references + + assert list((ROOT_DIR / "docs" / "papers").glob("*.pdf")) == [] + paper_index = read_text("docs/papers/README.md") + assert "does not itself grant downstream" in paper_index + assert "CC BY-NC-ND 4.0" in paper_index + + sha_pattern = re.compile(r"(? None: + """Keep the sole volatile evidence ledger aligned with its dated audit.""" + + evidence = read_text(DATED_EVIDENCE_APPENDIX) + assert "**Audit date:** 2026-08-11 (Asia/Seoul)" in evidence + snapshot = evidence.split("## Dated open-PR snapshot", 1)[1].split( + "## Dependency order from live refs", 1 + )[0] + normalized_snapshot = " ".join(snapshot.split()) + assert "Audited contributor head (pre-write)" in snapshot + assert "publishing this ledger necessarily advances" in normalized_snapshot + assert "not post-write current-head claims" in normalized_snapshot + observed = { + int(number) + for number in re.findall(r"^\| #(\d+) \|", snapshot, flags=re.MULTILINE) + } + assert observed == AUDITED_OPEN_PR_NUMBERS + inventory_rows = [ + line.split("|") + for line in snapshot.splitlines() + if re.match(r"^\| #\d+ \|", line) + ] + draft_count = sum( + columns[4].strip().split("/", 1)[0].strip() == "yes" + for columns in inventory_rows + ) + all_draft_claim = re.search(r"All (\d+) open PRs were Draft", snapshot) + partial_draft_claim = re.search( + r"(\d+) of (\d+) open PRs were Draft", snapshot + ) + if all_draft_claim is not None: + claimed_draft_count = claimed_total_count = int(all_draft_claim.group(1)) + else: + assert partial_draft_claim is not None + claimed_draft_count = int(partial_draft_claim.group(1)) + claimed_total_count = int(partial_draft_claim.group(2)) + assert claimed_total_count == len(inventory_rows) + assert claimed_draft_count == draft_count + assert "#80" not in snapshot + assert "#88" not in snapshot + for audited_head in ( + "ada372df205271c74ad095e898644588c7156075", # PR #109 + "8760993cb8262922a771948845c8dfd2afefb773", # PR #108 + "28088b9fc86d975b43637b7758d25e20d61c5786", # PR #107 + "828ca54f2b96a3bdd7adec24a26c0d8164df47d1", # PR #105 + "8453f082672d96b564ff2c32d028b11e05d8729f", # PR #104 + "2502915a8e90059074167e6306b47148a1d40fdc", # PR #99 + "73ed3a077f88a2f03cf734f1067bee2dcce2467f", # PR #94 + ): + assert audited_head in snapshot + + +def test_evidence_index_identifies_and_validates_the_latest_appendix() -> None: + """Make the latest volatile evidence discoverable without rewriting history.""" + + evidence_index = read_text("docs/evidence/README.md") + latest_name = Path(LATEST_EVIDENCE_APPENDIX).name + assert evidence_index.count(f"({latest_name})") == 1 + assert "Latest collected evidence" in evidence_index + + latest = read_text(LATEST_EVIDENCE_APPENDIX) + assert latest.startswith("# Continuation evidence audit — 2026-08-12\n") + assert "**Audit date:** 2026-08-12 (Asia/Seoul)" in latest + assert "**Document state:** `active_pr`" in latest + normalized_latest = " ".join(latest.split()) + assert ( + "The PR #105 row is a pre-write snapshot. Publishing this appendix " + "advances the PR #105 branch beyond the recorded row. The row is not " + "current-head evidence after publication and must be refetched." + in normalized_latest + ) + for heading in ( + "## Evidence identity rules", + "## Protected and dependency authority", + "## Open pull-request continuation snapshot", + "## Open issue continuation snapshot", + "## Operational and release acceptance", + "## Next acceptance boundary", + ): + assert heading in latest + + observed_prs = { + int(number) + for number in re.findall(r"^\| #(\d+) \|", latest, flags=re.MULTILINE) + } + assert observed_prs == AUDITED_OPEN_PR_NUMBERS + for status in ("exact_head_success", "blocked", "absent", "historical"): + assert f"`{status}`" in latest + + +def test_dated_central_prerequisite_snapshot_is_fail_closed() -> None: + """Keep incomplete central dependencies distinct from accepted authority.""" + + evidence = read_text(DATED_EVIDENCE_APPENDIX) + assert '.github PR #929 JSON repair: test-only RED' in evidence + assert '.github issue #907 wrapper repair: no completing PR' in evidence + assert "#906 had ten green" in evidence + assert "no formal review or qualifying approval" in evidence + assert "no production repair or associated workflow run" in evidence + assert "None of those states is protected integration" in evidence + + +def test_canonical_documentation_change_is_recorded_in_changelog() -> None: + """Keep the buyer-visible canonical documentation graph in release history.""" + + changelog = read_text("CHANGELOG.md") + assert ( + "Establish a canonical status-qualified product documentation graph" + in changelog + ) + assert "Add a canonical release, migration, and rollback guide" in changelog + assert "Replace legacy lab framing in the root README" in changelog + assert "Replace the competitor-centric README disclaimer" in changelog + assert "Remove the remaining stdlib-lab qualifier" in changelog + assert "Replace stale lab/prototype and internal-name language" in changelog + assert "Align Claude and conductor guidance" in changelog + assert "Status-qualify the analytics, REST API" in changelog + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit("Run with pytest so every documentation contract executes.") + + +def test_usage_evidence_adrs_define_one_mode_complete_contract() -> None: + """Keep every execution mode on one qualified usage-evidence contract.""" + + routing_adr = " ".join( + read_text("docs/adr/0005-sync-batch-pg-llm-batch.md").lower().split() + ) + evidence_adr = " ".join( + read_text("docs/adr/0006-honest-cost-and-benchmark-evidence.md").lower().split() + ) + for mode in ("sync completion", "batch retrieval", "passthrough", "route streaming"): + assert mode in routing_adr + for status in ("unknown", "not_recorded"): + assert chr(96) + status + chr(96) in routing_adr + assert "excluded from cost comparison" in routing_adr + + for dimension in ( + "account", + "service", + "upstream_api", + "model_name", + "team", + "group", + "company", + ): + assert chr(96) + dimension + chr(96) in evidence_adr + assert "mode-by-mode completeness tests" in evidence_adr + assert "writer and export path" in evidence_adr + + +def test_access_grant_model_is_directional_and_predecessor_bounded() -> None: + """Require consumer-to-producer access without bidirectional visibility.""" + + erd = " ".join(read_text("docs/ERD.md").split()) + for term in ( + "ACCESS_GRANT {", + "consumer_step_id", + "producer_step_id", + "authorized producer", + "earlier workflow step", + "does not grant bidirectional visibility", + ): + assert term in erd + assert "WORKFLOW_STEP }o--o{ WORKFLOW_STEP : exposes_by_access_list" not in erd + + +def test_planned_web_dependencies_are_not_presented_as_package_extras() -> None: + """Keep unshipped web-client frameworks explicitly planned.""" + + i18n = read_text("docs/i18n_design.md") + assert "i18next and React-admin are planned adoption candidates" in i18n + assert "i18next and React-admin are optional compatibility extras" not in i18n + + +def test_hybrid_llm_source_and_license_authority_are_linked() -> None: + """Link paper provenance separately from arXiv license evidence.""" + + paper_index = read_text("docs/papers/README.md") + references = read_text("docs/REFERENCES.md") + license_authority = "https://arxiv.org/abs/2404.14618" + assert license_authority in paper_index + assert license_authority in references + assert "https://openreview.net/forum?id=02f3mUtqnM" in paper_index + assert "https://openreview.net/forum?id=02f3mUtqnM" in references + + +def test_prd_and_trd_require_all_coverage_dimensions() -> None: + """Align product and technical release gates with ADR-0016.""" + + required_contract = "statement, branch, function, and line coverage" + for path in ( + "docs/PRD.md", + "docs/TRD.md", + "docs/adr/0016-complete-coverage-docstrings.md", + ): + assert required_contract in " ".join(read_text(path).split()) + + +def test_canonical_authority_state_cells_use_only_status_vocabulary() -> None: + """Reject descriptive prose in the canonical authority state column.""" + + index_text = read_text("docs/README.md") + rows = [ + line + for line in index_text.splitlines() + if line.startswith("| ") and not line.startswith("|---") + ] + authority_rows = [line for line in rows if "[" in line and "](" in line] + assert authority_rows + for row in authority_rows: + cells = [cell.strip() for cell in row.strip("|").split("|")] + assert len(cells) == 5 + state = cells[3].strip(chr(96)) + assert state in STATUS_VOCABULARY, row + + +def test_provider_failure_uml_separates_caller_and_provider_failures() -> None: + """Keep caller rejection terminal while documenting eligible provider failover.""" + + uml = read_text("docs/UML.md") + for line in ( + "alt caller validation error", + "Orchestrator-->>Orchestrator: Terminate without provider dispatch", + "alt transient provider failure", + "else permanent provider or configuration error", + "Orchestrator->>Fallback: Invoke eligible candidate without client retry", + ): + assert line in uml + + +def test_deployment_uml_keeps_credentials_behind_provider_adapter() -> None: + """Keep credential retrieval out of the orchestration-policy boundary.""" + + uml = read_text("docs/UML.md") + assert 'provider_adapter["Provider adapter"]' in uml + assert "policy --> provider_adapter" in uml + assert "provider_adapter --> kv" in uml + assert "policy --> kv" not in uml + + +def _assert_independent_review_evidence_fails_closed(decision: str) -> None: + """Assert that ADR-0010 retains every fail-closed review control.""" + + normalized = " ".join(decision.split()) + required_controls = ( + "`reviewDecision` must be `APPROVED` for the unchanged head", + ( + "A missing decision, `REVIEW_REQUIRED`, or `CHANGES_REQUESTED` " + "blocks the mutation even when branch protection currently allows " + "zero approvals" + ), + "An eligible independent non-author approval must also be present", + ( + "the aggregate field is evidence of the combined repository state, " + "not a substitute reviewer" + ), + ( + "A completed, successful, structured same-head Strix report is " + "separately required" + ), + ( + "Queued, in-progress, neutral/no-report, cancelled, skipped, absent, " + "or predecessor-head Strix states block merge" + ), + ( + "If the aggregate review state regresses or a required check becomes " + "incomplete after auto-merge is queued, automation disables that " + "queued mutation and starts exact-head verification again" + ), + ) + for required_control in required_controls: + assert required_control in normalized + + +def test_independent_review_evidence_fails_closed() -> None: + """Require canonical review evidence to reject incomplete aggregate gates.""" + + decision = read_text("docs/adr/0010-independent-review-and-evidence.md") + _assert_independent_review_evidence_fails_closed(decision) + + semantic_regressions = ( + ( + "blocks the mutation even when branch protection currently allows " + "zero approvals", + "is advisory when branch protection currently allows zero approvals", + ), + ( + "An eligible independent non-author approval must also be present", + "An eligible independent non-author approval is optional", + ), + ( + "structured same-head Strix report is separately required", + "structured same-head Strix report is advisory", + ), + ( + "block merge", + "may permit merge", + ), + ( + "automation disables that queued mutation", + "automation retains that queued mutation", + ), + ) + normalized = " ".join(decision.split()) + for required_control, weakened_control in semantic_regressions: + assert required_control in normalized + weakened = normalized.replace(required_control, weakened_control, 1) + with pytest.raises(AssertionError): + _assert_independent_review_evidence_fails_closed(weakened) + +def test_canonical_documentation_has_no_trailing_whitespace() -> None: + """Keep canonical Markdown compatible with diff-integrity gates.""" + + for path in REQUIRED_FILES: + document = read_text(path) + assert not document.endswith("\n\n"), f"{path}: final blank line" + for line_number, line in enumerate(document.splitlines(), start=1): + assert line == line.rstrip(), f"{path}:{line_number}" diff --git a/tests/test_documentation_current_truth.py b/tests/test_documentation_current_truth.py new file mode 100644 index 000000000..242881acc --- /dev/null +++ b/tests/test_documentation_current_truth.py @@ -0,0 +1,115 @@ +"""Reject stale implementation status and credential-authority claims in canonical docs.""" + +from pathlib import Path + +import pytest + + +ROOT_DIR = Path(__file__).resolve().parents[1] +CANONICAL_STATUS_PATHS = ( + "ARCHITECTURE.md", + "docs/PRD.md", + "docs/TRACEABILITY.md", + "docs/TRD.md", +) +CLOSED_UNMERGED_STACKS = (66, 82, 90, 94, 99, 113, 120) +OPEN_PARTIAL_STACKS = (111, 112, 114, 121) +PARTIAL_STATUS_MARKERS = ( + "partial", + "prototype", + "incomplete", + "blocker", + "not protected", + "no release authority", + "unprotected", +) + + +def _read(relative_path: str) -> str: + """Return one canonical repository document as UTF-8 text.""" + + return (ROOT_DIR / relative_path).read_text(encoding="utf-8") + + +def _matching_status_lines(pull_request_number: int) -> list[str]: + """Return canonical lines that name one pull request.""" + + return [ + line + for path in CANONICAL_STATUS_PATHS + for line in _read(path).splitlines() + if f"PR #{pull_request_number}" in line + ] + + +@pytest.mark.parametrize("pull_request_number", CLOSED_UNMERGED_STACKS) +def test_closed_unmerged_stack_is_superseded_not_active( + pull_request_number: int, +) -> None: + """Prevent closed-unmerged work from being promoted as a live implementation.""" + + matching_lines = _matching_status_lines(pull_request_number) + + assert matching_lines, ( + f"canonical docs must preserve the supersession boundary for PR " + f"#{pull_request_number}" + ) + assert all("`active_pr`" not in line for line in matching_lines) + assert any("`superseded`" in line for line in matching_lines) + + +@pytest.mark.parametrize("pull_request_number", OPEN_PARTIAL_STACKS) +def test_open_partial_stack_is_active_but_not_presented_as_complete( + pull_request_number: int, +) -> None: + """Keep open partial work visible without treating it as issue completion.""" + + matching_lines = _matching_status_lines(pull_request_number) + normalized_lines = [line.lower() for line in matching_lines] + + assert matching_lines, ( + f"canonical docs must preserve the open partial boundary for PR " + f"#{pull_request_number}" + ) + assert any("`active_pr`" in line for line in matching_lines) + assert all("closed-unmerged" not in line for line in matching_lines) + assert any( + marker in line + for line in normalized_lines + for marker in PARTIAL_STATUS_MARKERS + ), f"PR #{pull_request_number} must be qualified as partial or non-authoritative" + + +def test_reopened_nim_scaffold_is_superseded_without_false_closed_state() -> None: + """Keep the reopened #115 scaffold outside active authority without calling it closed.""" + + traceability = _read("docs/TRACEABILITY.md") + matching_lines = [ + line for line in traceability.splitlines() if "PR #115" in line + ] + + assert matching_lines + assert all("`active_pr`" not in line for line in matching_lines) + assert any("`superseded`" in line for line in matching_lines) + assert all("closed-unmerged" not in line for line in matching_lines) + assert any("open scaffold" in line for line in matching_lines) + + +def test_configured_postgres_kv_is_fail_closed_not_a_memory_fallback() -> None: + """Keep canonical config authority aligned with the active #96 composition root.""" + + architecture = _read("ARCHITECTURE.md") + technical_requirements = _read("docs/TRD.md") + normalized = " ".join(architecture.split()) + combined = " ".join(f"{architecture} {technical_requirements}".split()) + + assert ( + "An explicitly configured Postgres KV backend is authoritative and fails " + "closed with ConfigBackendUnavailableError" + in normalized + ) + assert "config and token-count adapters may silently fall back" not in combined + assert ( + "token counting may deliberately degrade to the documented heuristic" + in combined.lower() + ) diff --git a/tests/test_plugin_driven_artifacts.py b/tests/test_plugin_driven_artifacts.py index d64052556..06158981d 100644 --- a/tests/test_plugin_driven_artifacts.py +++ b/tests/test_plugin_driven_artifacts.py @@ -148,7 +148,7 @@ def test_figma_artifacts_are_recorded_without_code_connect() -> None: assert expected_text in artifacts -def test_ponytail_packaging_decision_keeps_commercial_product_unified() -> None: +def test_packaging_decision_keeps_commercial_product_unified() -> None: research = read_text("docs/library_research.md") for expected_text in [ @@ -157,10 +157,12 @@ def test_ponytail_packaging_decision_keeps_commercial_product_unified() -> None: "Do not split the", "Git submodule", "Extraction triggers", - "single-repo product", + "single-repository product", ]: assert expected_text in research + assert "Ponytail" not in research + def test_commercial_plugin_operating_model_defines_plugin_execution_scope() -> None: model = read_text("docs/commercial_plugin_operating_model.md") diff --git a/tests/test_repository_security_metadata.py b/tests/test_repository_security_metadata.py index b1c9ee785..b708eda9d 100644 --- a/tests/test_repository_security_metadata.py +++ b/tests/test_repository_security_metadata.py @@ -101,6 +101,145 @@ def test_security_policy_documents_reporting_and_automation(): assert "pinned to reviewed commit SHAs or hash-locked package requirements" in policy_text +def test_security_policy_documents_coordinated_disclosure_lifecycle(): + policy_text = read_text("SECURITY.md") + doctoring_text = read_text("docs/doctoring/security-disclosure-lifecycle.md") + + def section(document, heading): + start = document.index(heading) + len(heading) + end = document.find("\n## ", start) + return document[start:] if end == -1 else document[start:end] + + required_policy_tokens = [ + "## Supported Versions", + "## Scope", + "## Reporting a Vulnerability", + "## Coordinated Disclosure Lifecycle", + "## Safe Harbor and Research Boundaries", + "## Advisory and Release Evidence", + "latest supported release", + "GitHub Security Advisory", + "acknowledgement target", + "not a remediation SLA", + "CVE", + "Reporter credit", + "public issue", + "Do not include exploit details", + "ISO/IEC 29147:2018", + "ISO/IEC 30111:2019", + ] + for token in required_policy_tokens: + assert token in policy_text + + supported_versions = section(policy_text, "## Supported Versions") + assert "No stable release currently exists" in supported_versions + assert "`main` is not a supported release" in supported_versions + assert "version or release line" in supported_versions + + reporting = section(policy_text, "## Reporting a Vulnerability") + normalized_reporting = " ".join(reporting.split()) + assert "Remove credentials, personal data" in reporting + assert "Do not include exploit details, secrets, personal data" in reporting + assert "Before any stable release" in normalized_reporting + assert "private vulnerability reporting is enabled" in normalized_reporting + assert "security-notification recipients are configured" in normalized_reporting + assert "release authorization remains blocked" in normalized_reporting + assert "monitored alternative private contact" in normalized_reporting + + lifecycle = section(policy_text, "## Coordinated Disclosure Lifecycle") + lifecycle_stages = ( + "Receive and acknowledge", + "Validate and scope", + "Remediate and verify", + "Coordinate release", + "Publish evidence", + "Learn and prevent recurrence", + ) + lifecycle_positions = [lifecycle.index(stage) for stage in lifecycle_stages] + assert lifecycle_positions == sorted(lifecycle_positions) + + safe_harbor = section(policy_text, "## Safe Harbor and Research Boundaries") + for prohibited_activity in ( + "denial-of-service testing", + "social engineering", + "credential stuffing", + "destructive testing", + "high-volume automated probing", + ): + assert prohibited_activity in safe_harbor + + release_evidence = section(policy_text, "## Advisory and Release Evidence") + canonical_nonpassing_states = ( + "queued", + "pending", + "skipped-required", + "cancelled", + "failed", + "absent", + "stale-head", + "predecessor-head", + "author-only", + "status-only", + "synthetic-merge-only", + "rate-limited", + "infrastructure-only", + ) + assert "docs/RELEASE_GUIDE.md" in release_evidence + assert "exact integrated revision" in release_evidence + for state in canonical_nonpassing_states: + assert state in release_evidence + + required_doctoring_tokens = [ + "ISO/IEC 29147:2018", + "ISO/IEC 30111:2019", + "reviewed and confirmed", + "GitHub private vulnerability reporting", + "repository security advisory", + "NIST SP 800-218 Rev. 1", + "Initial Public Draft", + "Harold Booth", + "Michael Ogata", + "Karen Kent", + "Murugiah Souppaya", + "Donna Dodson", + "https://doi.org/10.6028/NIST.SP.800-218r1.ipd", + "APA 7", + ] + for token in required_doctoring_tokens: + assert token in doctoring_text + + doctoring_contract = section(doctoring_text, "## Repository contract") + assert "docs/RELEASE_GUIDE.md" in doctoring_contract + assert "exact integrated revision" in doctoring_contract + for state in canonical_nonpassing_states: + assert state in doctoring_contract + + +def test_agent_guidance_preserves_central_review_authority_and_nim_development_key(): + for guidance_path in ("AGENTS.md", "CLAUDE.md"): + guidance_text = read_text(guidance_path) + + assert "stays on **GitHub Models**" not in guidance_text + assert "stays on GitHub Models" not in guidance_text + assert "centrally governed" in guidance_text + assert "`NVIDIA_NIM_API_KEY`" in guidance_text + assert "`COPILOT_GITHUB_TOKEN`" in guidance_text + + +def test_agent_guidance_enforces_writer_lease_and_read_only_dependencies(): + required_tokens = ( + "one writer per repository branch", + "exact PR head and target blob SHA", + "read-only dependencies", + "write-capable agents", + "stale-head", + ) + for guidance_path in ("AGENTS.md", "CLAUDE.md"): + guidance_text = read_text(guidance_path) + for required_token in required_tokens: + assert required_token in guidance_text + + def test_database_design_avoids_plaintext_prompt_output_storage(): database_text = read_text("docs/database_design.sql") @@ -140,6 +279,9 @@ def test_security_tool_lockfile_uses_hash_pinning(): test_dependabot_tracks_actions_and_python_dependencies() test_codeowners_requires_repository_owner_review() test_security_policy_documents_reporting_and_automation() + test_security_policy_documents_coordinated_disclosure_lifecycle() + test_agent_guidance_preserves_central_review_authority_and_nim_development_key() + test_agent_guidance_enforces_writer_lease_and_read_only_dependencies() test_database_design_avoids_plaintext_prompt_output_storage() test_python_lockfile_uses_hash_pinning() test_security_tool_lockfile_uses_hash_pinning()