docs(research): trace review failures and paper provenance

.github/opencode/hourly-loop-prompt.md

@@ -9,6 +9,17 @@ completion, explicit user/operator cancellation, and the hosting platform's own
 execution contract are the only time-based termination authorities available to
 this agent. Never post intermediate progress reports.
 
+PR 0 only via merge or verified-successor full-delta inheritance. Never
+force-push, never close without evidence (user-explicit, no valid delta,
+malicious change, or verified complete inheritance only). Single-writer
+deltas are integrated, never discarded. Before and after long runs record
+`git rev-parse HEAD` and `git diff --stat origin/main...HEAD --
+contextual_orchestrator/ tests/`; docs-only drift does not invalidate code
+evidence. Prefer isolated worktrees, preserve live execution handles, and
+never rebase or push another session's branch. Synthetic recovery is unit
+evidence only; customer accuracy and decision latency require observed
+outcomes with declared denominators and uncertainty.
+
 Before changing code, read `docs/product_planning.md`, the applicable PRD in
 `docs/model-group-product-technical-spec.md`, and
 `docs/product-technical-gap-baseline.md`. Treat current files and exact GitHub

AGENTS.md

@@ -3,6 +3,15 @@
 Cross-agent conventions for `contextual-orchestrator`, readable by any coding
 agent (Claude, Codex, Cursor, opencode, …). Keep this file tool-agnostic.
 
+## Autonomous research handoff
+
+Read [the KPI runbook](docs/doctoring/autonomous_kpi_runbook.md) before numerical
+experiments and update its verified evidence before handoff. Choose KPI scope
+autonomously under `docs/analytics_spec.md`. Preserve live execution handles;
+high host load and silent numerical work are not proof of deadlock. Synthetic
+recovery is unit evidence, not customer accuracy. Break owner/consumer release
+cycles with isolated exact-revision contracts, never production source copies.
+
 <!-- BEGIN cwl-agent-guidance -->
 ## Agent guidance (CWL governance)
 

CLAUDE.md

@@ -4,6 +4,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Read AGENTS.md first
 
+For autonomous experiments, also read and maintain
+[the single KPI runbook](docs/doctoring/autonomous_kpi_runbook.md).
+It records commands, environment limits, failed interpretations, and evidence
+boundaries. Do not restart a live numerical test because polling is silent or
+claim customer improvement from synthetic recovery. Use the independently
+verifiable owner/consumer sequence recorded there instead of waiting for all
+foundation releases before developing a port.
+
 Equivalent model-group endpoints may race only through the normalized, explicit
 endpoint-equivalence contract. Preserve modality validation, bounded concurrency,
 deadline, cancellation/drain provenance, and honest duplicate-cost evidence.

contextual_orchestrator/benchmark_priors.py

@@ -1,40 +1,37 @@
 """Benchmark-quality priors for the model-group Beta ledgers.
 
-Two public measurements feed each member's prior success probability:
-LMSYS Chatbot Arena Elo (Bradley–Terry rating scale) and Artificial
-Analysis' Quality Index. Both are *published measurements*; this module
-never invents a numeric weight. Everything else is derived from either
-
-1. those measurements themselves,
-2. an existing repository constant (``model_group``'s Laplace prior
-   budget), or
-3. arithmetic over the items above.
+Legacy heuristic priors combine shipped Chatbot Arena ratings and Artificial
+Analysis Quality Index values. Their snapshot provenance has not been verified
+against archived source data. The transform below is deterministic but is not a
+calibrated probability of task success or an identified psychometric estimate.
 
 Derivation contract (auditable, deterministic):
 
 - Each shipped rating ``r_i`` is centered on the median of the shipped
   set and scaled by the set's own median absolute deviation (MAD),
   giving ``z_i = (r_i - median) / MAD_i``.
-- The two instruments are averaged after normalization (they measure
-  overlapping-but-distinct constructs; equal weight is the maximum
-  entropy choice across exactly two sources, not a tuned parameter).
-- ``p_hat = logistic(z)`` is then a posterior-style membership value in
-  ``(0, 1)`` measured from ratings alone.
+- The two instruments are averaged after normalization. Equal weighting is
+  an implementation choice, not a fitted weight justified by these references.
+- ``p_hat = logistic(z)`` maps the composite to ``(0, 1)``. That range alone
+  does not establish posterior or predictive calibration.
 - The prior is *mass preserving*: ``(alpha0, beta0)`` splits the exact
   unobserved-evidence budget that ``model_group`` already spends on any
   unknown member (its Laplace counts), so a known member never receives
   more evidence than an unknown one — it only receives that identical
   budget distributed according to measurement instead of uniformly.
 
-Failure denominator: members absent from every shipped instrument fall
-back to the unchanged repository Laplace prior.
+Members missing from either instrument fall back to the existing Laplace prior.
+Neither the fixed evidence budget nor that fallback proves validity. Replacing
+this legacy behavior requires observed-task calibration and release evidence;
+do not treat these values as customer accuracy evidence.
 
 References (APA 7th):
     Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete
         block designs: I. The method of paired comparisons. *Biometrika,
         39*(3/4), 324–345. https://doi.org/10.1093/biomet/39.3-4.324
-    Chiang, W., Zheng, L., Ma, Z., Li, Y., Sheng, Z., Wu, X., ... Zhang,
-        H. (2024). *Chatbot Arena: An open platform for evaluating LLMs
+    Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T.,
+        Li, D., Zhang, H., Zhu, B., Jordan, M. I., Gonzalez, J. E., &
+        Stoica, I. (2024). *Chatbot Arena: An open platform for evaluating LLMs
         by human preference* [Preprint]. arXiv.
         https://doi.org/10.48550/arXiv.2403.04132
 """
@@ -123,7 +120,7 @@ def _normalized_membership(name: str) -> float | None:
 
 
 def measured_quality_probability(member_id: str) -> float | None:
-    """Return the measurement-derived prior success probability, if known."""
+    """Return the legacy heuristic prior fraction, if both scores are known."""
     lowered = member_id.lower()
     for key in _ARENA_ELO:
         if key in lowered:
@@ -134,7 +131,7 @@ def measured_quality_probability(member_id: str) -> float | None:
 
 
 def resolve_quality_prior(member_id: str) -> tuple[float, float]:
-    """Resolve the benchmark-measured ``(alpha, beta)`` prior for one member.
+    """Resolve the legacy benchmark-derived ``(alpha, beta)`` prior.
 
     Unknown members receive the repository's unchanged Laplace pair, so
     behaviour for unmeasured identifiers is bit-for-bit the pre-existing

docs/analytics_spec.md

@@ -140,9 +140,39 @@ never present it as an all-request guarantee or encode missing durations as zero
 Declare the observation window, quantile method, uncertainty method, and workload
 before comparing policies. A faster failed decision is not a quality improvement.
 
+### Autonomous experiment targets
+
+These are engineering acceptance targets selected on 2026-09-09, not measured
+results or literature-derived constants. Do not ask the user to choose their
+scope. Baseline and candidate must use the same declared population, endpoint
+mix, task rubric, model versions, resource budget, and failure accounting.
+
+| Outcome | Target | Guardrail |
+| --- | --- | --- |
+| Delivered-correct fraction over all accepted requests | At least +1 percentage point versus baseline, with a 95% confidence interval for the difference wholly above zero. | Independently adjudicated observed outcomes; paired or randomized design declared before evaluation; no silent exclusion of failed delivery. |
+| Routing-decision p95 | At most 20 ms and at least 10% lower than baseline, with the 95% interval for the candidate/baseline ratio wholly below 1. | Include selection and durable acknowledgement; preserve workload and failure accounting. This is not the full-page latency SLO. |
+| Numerical parameter recovery | No regression in family-wise aligned RMSE under the declared numerical tolerance. | Known-truth unit tests only; never substitute for observed customer outcomes. |
+
+The end target requires both customer accuracy and decision-latency criteria.
+Intermediate changes may advance one while preserving the other's established
+baseline; they must not be labelled completion of both. If non-regression cannot
+be established, keep the candidate experimental and leave production unchanged.
+Use a fresh holdout for confirmation after adaptive experiment selection, or a
+predeclared sequential inference procedure; repeated inspection of an ordinary
+95% interval is not a stopping rule. Determine sample size from pilot variance,
+the +1-point effect target, and declared power before confirmation, retaining
+task/model/time clustering. Do not reduce sample size after seeing results.
+
+Execution and evidence handoff: [autonomous KPI runbook](doctoring/autonomous_kpi_runbook.md).
+
 ## Commercial Due-Diligence KPIs
 
-These metrics support the KRW 2,000,000,000 commercial-readiness review. The
+The active goal uses a USD 20,000,000,000 sale-quality ambition. This is an
+aspirational quality target, not a measured valuation or a signed transaction.
+The KRW 2,000,000,000 references in the legacy metrics below describe the
+earlier commercial-readiness review; they must not substitute for the active
+goal or be treated as its currency conversion. Accuracy and decision-latency
+acceptance still require the observed evidence specified above. The
 `evidence_type` column is mandatory so measured local evidence is never mixed
 with proposed production targets.
 

docs/benchmarks/2026-08-11-polytomous-llm-judge.md

@@ -3,6 +3,12 @@
 Status: exploratory integration evidence; not a claim that the local judge is
 unbiased or that this single case is sufficient for IRT estimation.
 
+Evidence classification (2026-09-09 audit): these historical model calls use
+constructed release-plan cases. They are not a probability sample of customer
+requests and must not supply the observed delivered-correct KPI or parameter
+recovery evidence. Tables below are historical reports, not freshly reproduced
+raw-artifact verification.
+
 ## Setup
 
 The path under test was:
@@ -137,7 +143,8 @@ parser rejected the model response; it was never repaired or accepted.
 | liked | `invalid` / `0.7500; yes` / `0.8333; yes` | `0.5000; no` / `0.0000; no` / `invalid` |
 | disliked | `0.5000; no` / `0.7500; yes` / `0.8333; yes` | `invalid` / `invalid` / `invalid` |
 
-The good plan parsed in 11/18 calls and was accepted in 5/11 parsed calls;
+The good plan parsed in 11/18 calls and was accepted in 5/11 parsed calls
+(45.45%, conditional on parsing), or 5/18 of all attempted calls (27.78%);
 the seven failures were five invalid JSON responses, one out-of-range category,
 and one non-monotone threshold vector. The unsafe plan parsed in all 18 calls,
 scored `0.0000` in every case, and was accepted zero times. Direct judging