Support local MLX orchestration and audited model judgment

.adr-config.yml

@@ -0,0 +1,6 @@
+project_slug: contextual-orchestrator
+owner: ContextualWisdomLab
+default_status: proposed
+decision_id_format: NNNN
+template_source: madr-v4
+last_decision_id: 0008

.github/workflows/fuzz.yml

@@ -83,6 +83,9 @@ jobs:
       - name: Fuzz orchestration engine
         run: python fuzz/fuzz_orchestration.py -max_total_time=${FUZZ_SECONDS} -artifact_prefix=crash- fuzz/corpus/orchestration
 
+      - name: Fuzz model-judge response parser
+        run: python fuzz/fuzz_model_judge.py -max_total_time=${FUZZ_SECONDS} -artifact_prefix=crash- fuzz/corpus/judge
+
       - name: Upload crash artifacts
         if: failure()
         uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # actions/upload-artifact@v5

CLAUDE.md

@@ -44,8 +44,9 @@ python fuzz/fuzz_request_body.py -max_total_time=60 fuzz/corpus/request_body
 python -m contextual_orchestrator "your prompt" --agents examples/agents.mock.json
 
 # Serve the OpenAI-compatible API + /admin console
-export CONTEXTUAL_ORCHESTRATOR_TOKEN="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
-python -m contextual_orchestrator --serve --agents examples/agents.mock.json --port 8000
+local_token="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
+python -m contextual_orchestrator --serve --agents examples/agents.mock.json --port 8000 \
+  --auth-token "$local_token"
 
 # Loopback-only local dev server (auth disabled; loopback only)
 ./.superset/run.sh

Dockerfile

@@ -26,4 +26,4 @@ HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
   CMD ["python", "-c", "import urllib.request,os;urllib.request.urlopen(f'http://127.0.0.1:{os.environ.get(\"PORT\",\"8000\")}/healthz', timeout=2)"]
 
 # --allow-public-bind: 컨테이너 내부 0.0.0.0 바인딩 필요(외부 노출은 호스트 포트 매핑이 결정)
-CMD ["sh", "-c", "python -m contextual_orchestrator --serve --agents \"$AGENTS_FILE\" --host 0.0.0.0 --port \"$PORT\" --allow-public-bind"]
+CMD ["sh", "-c", "python -m contextual_orchestrator --serve --agents \"$AGENTS_FILE\" --host 0.0.0.0 --port \"$PORT\" --allow-public-bind --auth-token-key CONTEXTUAL_ORCHESTRATOR_TOKEN"]

README.md

@@ -17,8 +17,9 @@ python -m contextual_orchestrator "Summarize why model orchestration helps long
 Run the OpenAI-compatible subset:
 
 ```bash
-export CONTEXTUAL_ORCHESTRATOR_TOKEN="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
-python -m contextual_orchestrator --serve --agents examples/agents.mock.json --port 8000
+local_token="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
+python -m contextual_orchestrator --serve --agents examples/agents.mock.json --port 8000 \
+  --auth-token "$local_token"
 ```
 
 Admin console:
@@ -29,14 +30,15 @@ http://127.0.0.1:8000/admin
 
 ```bash
 curl -s http://127.0.0.1:8000/v1/chat/completions \
-  -H "authorization: Bearer $CONTEXTUAL_ORCHESTRATOR_TOKEN" \
+  -H "authorization: Bearer $local_token" \
   -H 'content-type: application/json' \
   -d '{"model":"contextual-orchestrator","messages":[{"role":"user","content":"Analyze this code review task and verify the answer."}]}' | jq .
 ```
 
 HTTP serving is hardened for local lab use:
 
-- `/admin`, `/admin/state`, `/api/v1/*`, and `/v1/chat/completions` require a Bearer token. Use `--admin-token` and `--inference-token` to separate operator and runtime access, or `--auth-token` / `CONTEXTUAL_ORCHESTRATOR_TOKEN` for one local-development token.
+- `/admin`, `/admin/state`, `/api/v1/*`, and `/v1/chat/completions` require a Bearer token. Use `--admin-token-key` and `--inference-token-key` to resolve split tokens from the KV, or `--auth-token-key` for one token. Explicit `--auth-token`/split-token values are local-development escape hatches; the CLI no longer reads auth secrets from environment variables.
+- A production deployment that uses the ecosystem identity plane must inject a reviewed `bearer_verifier` into `SecurityConfig` to validate Keyverse-issued OIDC tokens (issuer, audience, signature, expiry, and scope). The core does not hand-roll JWT parsing or hold Keycloak admin credentials; a static bearer token is not a Keyverse integration.
 - Binding to `0.0.0.0` or `::` requires `--allow-public-bind`.
 - JSON request bodies, chat message roles, orchestration modes, body sizes, request rate, and concurrent run counts are validated before orchestration runs.
 - Full orchestration traces are not returned by default. Set `include_orchestration_trace: true` per chat request or start with `--expose-trace-by-default` when the caller is trusted.
@@ -60,6 +62,24 @@ Use real workers by replacing `mock://` agents with OpenAI-compatible endpoints.
 }
 ```
 
+For a local `mlx-lm` OpenAI-compatible server, use the explicit `mlx://` scheme. It is loopback-only, does not require a credential, and is translated to HTTP only after the loopback check:
+
+```json
+{
+  "agents": [
+    {
+      "id": "local_fast_agent",
+      "model": "mlx-community/llama-3.2-3b-instruct-4bit",
+      "base_url": "mlx://127.0.0.1:8080/v1",
+      "provider_name": "mlx-lm",
+      "tags": ["reasoning", "coding", "verification"]
+    }
+  ]
+}
+```
+
+Run an evaluation against that server with `--temperature 0` for repeatable judging. For reasoning-capable mlx models, pass `--chat-template-args '{"enable_thinking":false}'` when a short structured judge response is required. `--local-concurrency N` enables bounded concurrent local batch requests; keep interactive route/conduct requests on the default sequential path.
+
 The agent pool is manageable at runtime: `POST`/`PATCH`/`DELETE` on `/api/v1/agent_pools/default/worker_agents[/{id}]` add, govern, and remove model-group members. Pass `--agents-db PATH` (or `CONTEXTUAL_ORCHESTRATOR_AGENTS_DB`) to persist those changes to a stdlib sqlite file — stored changes overlay the seed agents file at startup, and removals write disabled tombstones so they survive restarts; without it the pool is in-memory as before.
 
 Seed the credential into the KV once at bootstrap:
@@ -68,6 +88,12 @@ Seed the credential into the KV once at bootstrap:
 echo "$OPENAI_API_KEY" | python -m contextual_orchestrator register-credential --name OPENAI_API_KEY --value-stdin
 ```
 
+For a persistent KV-backed server token, seed a credential such as
+`CONTEXTUAL_ORCHESTRATOR_TOKEN` and start with
+`--auth-token-key CONTEXTUAL_ORCHESTRATOR_TOKEN`. The in-memory credential
+backend is process-local and is suitable only for tests; production auth
+registration and OIDC client secrets belong to the deployment/KV boundary.
+
 Non-mock providers must use `https://` URLs and a **resolvable KV credential** — a non-mock agent whose credential is missing raises `NotConfigured` rather than falling back to an environment variable. The runtime blocks loopback, private, link-local, multicast, and reserved provider addresses before sending a key. Set `CONTEXTUAL_ORCHESTRATOR_ALLOWED_PROVIDER_HOSTS` to a comma-separated host allowlist when only approved model gateways should be reachable. External calls use a timeout and default output token cap.
 
 > The legacy `api_key_env` field is still accepted for back-compat, but its value is now treated as the **credential name** in the KV, not as an environment variable to read. This supersedes the old `api_key_env` env pattern.
@@ -125,7 +151,7 @@ Local spend observability, aggregated from in-memory workflow runs. It is honest
 
 ```bash
 curl -s http://127.0.0.1:8000/api/v1/spend_analytics/latest \
-  -H "authorization: Bearer $CONTEXTUAL_ORCHESTRATOR_TOKEN" | jq '.totals, .by_model, .budget'
+  -H "authorization: Bearer $local_token" | jq '.totals, .by_model, .budget'
 ```
 
 - **Tokens.** `by_model[].output_tokens` uses the provider-reported `usage.completion_tokens` when a real worker returns it, and falls back to a `~4 chars/token` estimate otherwise. Each row carries `usage_source`: `reported` (all steps reported), `mixed`, or `estimated`. `estimated_output_tokens` is always the estimate, kept alongside for comparison. `measurement_status` is `local_runtime_estimate`, not production telemetry.

contextual_orchestrator/__main__.py

@@ -7,11 +7,26 @@
 import os
 import sys
 
-from .credentials import register_credential
+from .credentials import get_credential, register_credential
 from .orchestrator import ModelClient, TaskOrchestrator, load_agents
 from .server import SecurityConfig, serve
 
 
+DEFAULT_AUTH_TOKEN_KEY = "CONTEXTUAL_ORCHESTRATOR_TOKEN"
+DEFAULT_ADMIN_TOKEN_KEY = "CONTEXTUAL_ORCHESTRATOR_ADMIN_TOKEN"
+DEFAULT_INFERENCE_TOKEN_KEY = "CONTEXTUAL_ORCHESTRATOR_INFERENCE_TOKEN"
+
+
+def _resolve_auth_token(explicit: str, credential_name: str) -> str:
+    """Resolve a server bearer token from an explicit local value or the KV."""
+    if explicit:
+        return explicit
+    token = get_credential(credential_name)
+    if not token:
+        raise ValueError(f"server auth credential '{credential_name}' is not configured in the KV")
+    return token
+
+
 def _register_credential_command(argv: list[str]) -> None:
     """Bootstrap: read a deploy-time secret and store it in the KV credential registry.
 
@@ -70,9 +85,15 @@ def main() -> None:
     parser.add_argument("--serve", action="store_true", help="Run the chat completions HTTP server.")
     parser.add_argument("--host", default="127.0.0.1")
     parser.add_argument("--port", type=int, default=8000)
-    parser.add_argument("--auth-token", default=os.environ.get("CONTEXTUAL_ORCHESTRATOR_TOKEN", ""))
-    parser.add_argument("--admin-token", default=os.environ.get("CONTEXTUAL_ORCHESTRATOR_ADMIN_TOKEN", ""))
-    parser.add_argument("--inference-token", default=os.environ.get("CONTEXTUAL_ORCHESTRATOR_INFERENCE_TOKEN", ""))
+    parser.add_argument("--auth-token", default="", help="Explicit local-development bearer token; prefer a KV token name.")
+    parser.add_argument("--admin-token", default="", help="Explicit local-development admin token; prefer a KV token name.")
+    parser.add_argument("--inference-token", default="", help="Explicit local-development inference token; prefer a KV token name.")
+    parser.add_argument("--auth-token-key", default=None,
+                        help="KV credential name for the single server bearer token.")
+    parser.add_argument("--admin-token-key", default=None,
+                        help="KV credential name for the admin bearer token.")
+    parser.add_argument("--inference-token-key", default=None,
+                        help="KV credential name for the inference bearer token.")
     parser.add_argument("--allow-public-bind", action="store_true")
     parser.add_argument("--insecure-disable-auth", action="store_true", help="Deprecated; API auth is always required.")
     parser.add_argument("--expose-trace-by-default", action="store_true")
@@ -82,8 +103,14 @@ def main() -> None:
                         help="Optional sqlite path so runtime agent-pool changes (add/patch/remove) survive restarts.")
     parser.add_argument("--provider-ca-bundle", default=os.environ.get("CONTEXTUAL_ORCHESTRATOR_PROVIDER_CA_BUNDLE") or None,
                         help="Path to a CA bundle used to verify provider TLS (e.g. a corporate gateway root).")
-    parser.add_argument("--insecure-skip-tls-verify", action="store_true",
-                        help="Dev only: do not verify provider TLS certificates (insecure).")
+    parser.add_argument("--temperature", type=float, default=0.2,
+                        help="Default provider sampling temperature (default: 0.2).")
+    parser.add_argument("--max-output-tokens", type=int, default=2048,
+                        help="Default provider output token cap (default: 2048).")
+    parser.add_argument("--local-concurrency", type=int, default=1,
+                        help="Concurrent requests for explicit mlx:// local batch work (default: 1).")
+    parser.add_argument("--chat-template-args", type=json.loads, default={},
+                        help="JSON kwargs forwarded to local mlx-lm chat templates, e.g. '{\"enable_thinking\":false}'.")
     parser.add_argument("--budget-max-output-tokens", type=int, default=None,
                         help="Refuse new runs once estimated/reported output tokens reach this cap (default: no cap).")
     parser.add_argument("--budget-max-cost-usd", type=float, default=None,
@@ -94,7 +121,13 @@ def main() -> None:
                         help="Measure orchestration vs a single-worker baseline on these prompts and print the report.")
     args = parser.parse_args()
 
-    client = ModelClient(ca_bundle=args.provider_ca_bundle, verify_tls=not args.insecure_skip_tls_verify)
+    client = ModelClient(
+        ca_bundle=args.provider_ca_bundle,
+        temperature=args.temperature,
+        max_output_tokens=args.max_output_tokens,
+        local_concurrency=args.local_concurrency,
+        chat_template_args=args.chat_template_args,
+    )
     orchestrator = TaskOrchestrator(
         load_agents(args.agents),
         client=client,
@@ -110,23 +143,44 @@ def main() -> None:
         return
 
     if args.serve:
-        if not (args.auth_token or args.admin_token or args.inference_token):
-            parser.error(
-                "--serve requires --auth-token, split --admin-token/--inference-token, "
-                "or matching CONTEXTUAL_ORCHESTRATOR_* environment variables"
-            )
-        if not args.auth_token and (args.admin_token or args.inference_token) and not (
-            args.admin_token and args.inference_token
+        single_requested = bool(args.auth_token or args.auth_token_key)
+        split_requested = bool(
+            args.admin_token or args.inference_token or args.admin_token_key or args.inference_token_key
+        )
+        if single_requested and split_requested:
+            parser.error("choose either --auth-token or the split --admin-token/--inference-token mode")
+        if split_requested and not (
+            (args.admin_token or args.admin_token_key) and (args.inference_token or args.inference_token_key)
         ):
             parser.error("split token mode requires both --admin-token and --inference-token")
+        try:
+            auth_token = (
+                _resolve_auth_token(args.auth_token, args.auth_token_key or DEFAULT_AUTH_TOKEN_KEY)
+                if not split_requested
+                else ""
+            )
+            admin_token = (
+                _resolve_auth_token(args.admin_token, args.admin_token_key or DEFAULT_ADMIN_TOKEN_KEY)
+                if split_requested
+                else ""
+            )
+            inference_token = (
+                _resolve_auth_token(args.inference_token, args.inference_token_key or DEFAULT_INFERENCE_TOKEN_KEY)
+                if split_requested
+                else ""
+            )
+        except ValueError as exc:
+            parser.error(str(exc))
+        if not (auth_token or admin_token or inference_token):
+            parser.error("--serve requires a KV auth credential or explicit local token")
         serve(
             orchestrator,
             host=args.host,
             port=args.port,
             security=SecurityConfig(
-                auth_token=args.auth_token,
-                admin_token=args.admin_token,
-                inference_token=args.inference_token,
+                auth_token=auth_token,
+                admin_token=admin_token,
+                inference_token=inference_token,
                 allow_public_bind=args.allow_public_bind,
                 expose_trace_by_default=args.expose_trace_by_default,
             ),