From b183eb25a83adb38b527bf1c07b72898e08689e0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 08:44:08 +0000 Subject: [PATCH 01/50] docs(agents): add CI failure-signature triage catalog and index the know-how stores Records seven recurring red-check signatures across .github, noema and contextual-orchestrator as a string-matchable triage catalog, so an agent looking at a failed required check can decide what to do without re-investigating a known mechanism. Every entry was verified against the code on origin/main before being written; the file:line citations are the evidence. Two of the catalogued signatures look like a defect in the PR under review and are not: the opencode-review wait-for-verdict fail-closed, and strix running stale trusted source because strix.yml resolves trusted_ref from workflow_sha, which GitHub pins at run-creation time rather than execution time. Adversarial verification changed three of the seven claims before they were written down, which is why they are recorded here rather than in a session's memory: nothing re-runs a failed noema-review automatically (the scheduler re-runs Strix only), an expired NIM cost window no longer fails PR tests since the autouse fixture landed in #1073, and "behind main" is not a safe diagnosis because main is frequently the broken side. Also closes a know-how propagation gap. .jules/bolt.md and .jules/sentinel.md carry the accumulated performance and security learnings for scripts/ci/, but were reachable only from CLAUDE.md, so no non-Claude agent had a path to them. AGENTS.md is the tool-agnostic entry point every agent is required to read, so it now indexes both stores plus this catalog. Adds a verification-discipline entry for working-tree staleness: main advances roughly every 20 minutes, one session's .github checkout was measured 218 commits behind on 2026-09-05, and reasoning from it re-derived an already-fixed bug and cited lines that do not exist on main. Documentation only. Full suite 2883 passed / 1 skipped, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- AGENTS.md | 35 +++ CLAUDE.md | 12 + docs/doctoring/ci-failure-signature-triage.md | 265 ++++++++++++++++++ 3 files changed, 312 insertions(+) create mode 100644 docs/doctoring/ci-failure-signature-triage.md diff --git a/AGENTS.md b/AGENTS.md index 881e3aabf5..c677449c05 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -99,3 +99,38 @@ history, never the organization's actual state. conclusions.** Sessions here share a model and tend to share blind spots. A read-only `codex exec -s read-only -C ""` pass has already caught a factual error in this very section that same-family review missed. +- **Your working tree is not evidence until you prove it is current.** A + long-running session's checkout drifts while `main` advances roughly every + 20 minutes; on 2026-09-05 one session's `.github` tree was 218 commits + behind, and from it re-derived an already-fixed bug and cited line numbers + that do not exist on `main`. At every wake, and before reading any file you + will reason about, run + `git fetch origin main && git rev-list --count HEAD..origin/main`. If that is + non-zero, read files as `git show origin/main:` or in a fresh + `git worktree add origin/main`, and never quote a line number you did + not read from `origin/main`. + +## Where accumulated know-how lives + +Durable agent learnings are recorded in this repository, not in any session's +private memory. Read the relevant store before working on the surface it +covers — these records exist so the same class of bug is not reintroduced, and +they only pay off when an agent other than their author reads them. + +- [`docs/doctoring/ci-failure-signature-triage.md`](docs/doctoring/ci-failure-signature-triage.md) + — string-matchable catalog of the recurring red-check signatures across the + three repositories (`opencode-review` wait-for-verdict, `strix` stale + `workflow_sha`, `noema-review` gateway 502, expired NIM cost evidence, + unscoped `pytest` collection, stale `head_sha` notifications, + queued-versus-failed), each with its verified mechanism, the action to take, + and the actions not to take. Also carries the research-grounding freshness + KPIs and their measured baseline. +- [`.jules/bolt.md`](.jules/bolt.md) — dated performance learnings for + `scripts/ci/` as `**Learning:**` / `**Action:**` pairs. Scan it before + optimizing anything on that surface. +- [`.jules/sentinel.md`](.jules/sentinel.md) — dated security learnings for the + same surface as `**Vulnerability:**` / `**Learning:**` / `**Prevention:**` + triples, covering HTML-comment breakout in serialized JSON, cross-language + redaction parity, `shell=True` "security theater", and SSRF via redirects. + Scan it before hardening anything on that surface. +- `docs/doctoring/` — per-incident and per-decision records generally. diff --git a/CLAUDE.md b/CLAUDE.md index 550ad5ce80..f5a017b938 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -177,3 +177,15 @@ repeatable compile command. longer than it is. Querying `status=success` and `status=failure` directly cuts through the churn to the most recent real conclusion of each kind. Those are historical signals about pipeline liveness only — they never substitute for exact-current-head evidence on the PR you are acting on. +- **Triage a red required check by signature, not by guesswork.** + [`docs/doctoring/ci-failure-signature-triage.md`](docs/doctoring/ci-failure-signature-triage.md) + catalogs the recurring failures across `.github`, `noema`, and `contextual-orchestrator` — the + `opencode-review` wait-for-verdict fail-closed, `strix` running stale trusted source because + `workflow_sha` is pinned at run-creation time, the `noema-review` gateway 502 that nothing re-runs + automatically, expired NIM cost evidence, `agent-review-runtime-quality` failing on tests the PR + never touched because its `pytest` call has no positional path, and queued-versus-failed. Each entry + gives the verified mechanism with `file:line`, what to do, and what not to do. Two of those + signatures look like a defect in your PR and are not. Read it before "fixing" a red check whose + cause you have not confirmed, and before merging `main` into a branch — main is frequently the + broken side. `AGENTS.md` also indexes `.jules/bolt.md` and `.jules/sentinel.md`, which are the + performance and security learnings for `scripts/ci/`. diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md new file mode 100644 index 0000000000..1c1333f362 --- /dev/null +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -0,0 +1,265 @@ +# CI failure-signature triage + +**Audience:** any agent (Claude, Codex, Grok, Gemini) looking at a red required check in +`ContextualWisdomLab/.github`, `ContextualWisdomLab/noema`, or +`ContextualWisdomLab/contextual-orchestrator` and deciding what to do in the next minute. + +**How to use it:** string-match the error you are looking at against a heading below, then follow +that section's **Do** / **Do not**. Every mechanism claim here was verified against the code on +`origin/main` before being written down; the file:line citations are the evidence, not decoration. +If a section's cited line no longer says what this document claims, the section is stale — fix it +here rather than working around it in a consuming repository. + +**Scope:** this is a *signature → action* catalog. Incident retrospectives and repair evidence live +in their own records, e.g. +[`noema-review-failure-retrospective-and-improvement-plan-20260903.md`](noema-review-failure-retrospective-and-improvement-plan-20260903.md), +[`startup-failure-and-strix-concurrency-20260904.md`](startup-failure-and-strix-concurrency-20260904.md), +and [`actions-queue-saturation-hourly-sweep.md`](actions-queue-saturation-hourly-sweep.md). + +Baseline for every claim below: `.github` `main@27d7331cc`, `contextual-orchestrator` `main@a080297d`, +`noema` `main@e1ac9d5`, all as of 2026-09-05. + +--- + +## 0. Before anything else: your local checkout is probably stale + +**Symptom.** None — that is the problem. Everything looks normal and your conclusions are wrong. + +**Mechanism.** Long-running agent sessions keep a working tree that was cloned once. `main` in this +organization advances roughly every 20 minutes. Measured 2026-09-05: `/home/user/.github` was **218 +commits behind** `origin/main` (`docs/product-technical-gap-baseline.md` 2615 lines locally vs 3234 +on main), and `/home/user/contextual-orchestrator` still showed `nim_benchmark.py` `valid_until_date` +`2026-09-04` when main already carried `2026-10-05`. Reasoning from that tree re-derives bugs that +are already fixed, and quoting its line numbers cites lines that do not exist on main. + +**Do.** At every wake, and before reading any file you intend to reason about or edit: + +```bash +git fetch origin main && git rev-list --count HEAD..origin/main +``` + +If the count is greater than zero, treat the working tree as **not evidence**. Read target files as +`git show origin/main:`, or work in a fresh `git worktree add origin/main`. Never quote a +line number you did not read from `origin/main`. + +**Do not.** Do not report a finding, open a PR, or file a gap entry whose evidence came only from an +unverified working tree. + +--- + +## 1. `opencode-review` fails with "No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head" + +**Symptom.** The required `opencode-review` check is red with exactly: +`No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head. The dispatch workflow will +rerun this failed job after publishing an authenticated exact-head verdict.` +(`.github/workflows/opencode-review.yml:488`). + +**Mechanism.** This is a designed fail-closed wait, not a defect in the PR's code. The same job first +fires a `repository_dispatch` carrying `required_run_id: $GITHUB_RUN_ID` +(`.github/workflows/opencode-review.yml:409`), then fails so it releases its runner instead of +polling. Later, `.github/workflows/opencode-review-dispatch.yml`'s "Wake exact-head required OpenCode +workflow" step (line 7532) calls `POST /actions/runs/{id}/rerun-failed-jobs` (line 7572) on that +exact run. + +**Do.** Check whether a newer `OpenCode Review Dispatch` run for this head is in flight; if yes, wait. +If none is, treat it as a real stall and inspect that run for: receipt-gate rejection +(`scripts/ci/opencode_review_receipt_gate.py:136` rejects fallback and model-unavailable approvals), a +missing `PR_REVIEW_MERGE_TOKEN` / `OPENCODE_APPROVE_TOKEN` wake credential +(`opencode-review-dispatch.yml:7549`), or a verdict published under a non-`opencode-agent` identity. + +**Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this +failure says nothing about it. + +--- + +## 2. `strix` fails within seconds with `AttributeError: ... no attribute 'asyncio'` or `ModuleNotFoundError: No module named 'httpx2'` + +**Symptom.** The required `strix` check dies in 0–2s: a traceback ending in +`AttributeError: 'function' object has no attribute 'asyncio'` **or** +`ModuleNotFoundError: No module named 'httpx2'`, then +`Strix run failed for model 'orchestrator/free' after Ns (exit code 1)`. + +**Mechanism.** `.github/workflows/strix.yml:398-400` resolves `trusted_ref` from `job.workflow_sha`, +and line 428 checks the entire trusted Strix source tree out of `ContextualWisdomLab/.github` at that +SHA. That is `.github` main HEAD as of **each run attempt's creation** — not its execution time. A +congestion-queued attempt therefore executes hours later against pre-fix source. The two errors are +different staleness windows: the `AttributeError` means the pin predates `87352d98` (#1783); the +`httpx2` import error means it predates `76969152` (#1851). + +**Do.** Confirm on current main that `scripts/ci/strix_timeout_compat.py` resolves the submodule via +`sys.modules["strix.interface.main"]` and that `requirements-strix-ci-hashes.txt` pins +`httpx2==2.12.0`. Then re-run the failed job: the re-run re-pins to main HEAD and runs the newer +`strix.yml`. + +**Do not.** Do not patch the consuming repository; its head SHA is not implicated. Do not assume the +re-run repeats identical source — a regression landing between your check and the re-run yields a +new, different error, so re-check main and re-run rather than concluding the fix failed. + +--- + +## 3. `noema-review` fails after ~30 minutes with `HTTP Error 502: Bad Gateway` and `caller attempts=1` + +**Symptom.** One job-log line: +`Noema gateway transport failed: HTTPError: HTTP Error 502: Bad Gateway; caller attempts=1, +duration=1814.3s, phase=response_error, served_model=` — observed durations 1800–2300s, on +`contextual-orchestrator` #1012 and #1043. + +**Mechanism.** Gateway-side, not the diff. `scripts/ci/noema_review_gate.py:1507-1674` sends exactly +one request with no retry loop, called once from `.github/actions/noema-review/two_phase.py:170`. The +gateway is a `127.0.0.1` sidecar vendored from the pinned SHA in +`scripts/ci/contextual_orchestrator_review_sidecar.sh:17` — never the PR head. A client-visible 502 is +raised only after `contextual_orchestrator/orchestrator.py:7774-7896` exhausted every candidate; +content-shaped rejections would surface as 400/413 +(`contextual_orchestrator/provider_errors.py:67-92`). + +**Do.** Re-run the failed `noema-review` job by hand. **Nothing re-runs it automatically** — +`scripts/ci/pr_review_merge_scheduler_core.py:3746` re-runs Strix only, and +`scripts/ci/noema_review_handoff.py`'s dispatch requires a reusable exact-head OpenCode approval. + +**Do not.** Do not blame `served_model`; it is merely the last candidate that failed +(`orchestrator.py:7812`). Do not diagnose it as a provider hang specifically — a 502 here equally +covers upstream 5xx, TLS, DNS and connection failures. Do not add `timeout-minutes` to +`.github/workflows/noema-review.yml`; its absence is deliberate. Do not edit the PR under review. + +--- + +## 4. `reviewed NVIDIA hosted-endpoint cost evidence expired` in contextual-orchestrator + +**Symptom.** `BenchmarkContractError: reviewed NVIDIA hosted-endpoint cost evidence expired; +re-review official terms`. + +**Mechanism.** `contextual_orchestrator/nim_benchmark.py:113-129` holds `ACTUAL_COST_EVIDENCE` +(`reviewed_at_date` / `valid_until_date`); `_require_current_actual_cost_evidence` (lines 2550-2571) +compares `today or datetime_module.date.today()` against it and fails closed. It is reached only from +`run_benchmark`'s live path (line 3181). Since #1073 (main `a080297`), +`tests/test_nim_benchmark.py:48-53` has an autouse fixture pinning the dates to +`2000-01-01` / `2999-12-31`, so an expired window **no longer fails PR tests**. Before that fixture it +failed five tests on every open PR regardless of diff, via +`contextual-orchestrator/.github/workflows/security.yml:60-62`. + +**Do.** Confirm the autouse fixture still exists (open PR #1070 proposes narrowing it to opt-in). +Otherwise re-review and update both dates together. + +**Do not.** Do not bump `valid_until_date` without that primary-source review — nothing in the code +detects a bare bump, so a date extension silently converts reviewed evidence into fabricated evidence. +Do not read the dates in `tests/test_nim_benchmark_release_acceptance.py` as production evidence; they +are synthetic pricing-scenario fixtures exercised with an injected `today=`, and they must be excluded +from any expired-evidence metric. + +--- + +## 5. `agent-review-runtime-quality` fails on `tests/` files the PR never touched + +**Symptom.** In `.github`, the `agent-review-runtime-quality` check fails inside the step +"Verify scheduler and contextual-orchestrator review-repair contracts", with failures in files your +diff never touched — e.g. `tests/test_pr_review_merge_scheduler.py` or +`tests/test_hourly_review_repair_callers.py`. This happens even on a documentation-only PR. + +**Mechanism.** `.github/workflows/agent-review-runtime-quality-ci.yml:371-381` runs +`python -m pytest -q --cov=... --cov-branch --cov-fail-under=100` with **no positional path**, and +`pyproject.toml` sets no `testpaths` (only `pythonpath = ["."]`), so collection covers every file +under `tests/`. Any red test in `tests/` fails your PR. The step is gated on +`if: steps.affected_suites.outputs.review_repair == 'true'`, which several documentation paths set +(`docs/product-technical-gap-baseline.md`, `docs/automation/hourly-review-repair.md`, +`docs/doctoring/*-hourly-review-caller.md`). The same unscoped pattern exists in +`agent-mention-router-quality-ci.yml:108` and `repository-metadata-reconcile.yml:100`. + +**Do.** Check whether current `main`'s tip passes those same tests. If main is green, merge current +main into the branch, run the FULL suite locally, and push the merge commit. If main is red, land the +repair on main first — precedent: #1877, #1883. + +**Do not.** Do not assume "behind main" is the cause: **main is often the broken side**, and merging a +broken main into your branch then loops. Never force-push. Do not treat +`tests/test_docs_only_pr_runner_admission.py` as this signature; it is named explicitly by the strix +step. + +--- + +## 6. A `check_run` failure whose `head_sha` no longer matches the live PR head + +**Symptom.** A `check_run` / `check_suite` failure notification, or a scheduler log line such as +`current head has no OpenCode approval; branch is outdated before review dispatch` or +`current-head OpenCode review requested changes; branch is outdated before re-review`, references a +SHA that is no longer `pull_request.head.sha`. + +**Mechanism.** `scripts/ci/pr_review_merge_scheduler_core.py`'s `inspect_pr` calls +`request_branch_update` on four paths: approved head (lines 4648-4685), review requested changes +(4382-4395), no current-head approval before review dispatch (4774-4792), and `restamp_head` +(4687-4730). The failed-check block at line 4563 sits inside `if current_head_approved:`, so failing +checks do **not** stop the second and third paths. `branch_outdated_by_base` (1612-1617) also fires on +the REST compare `behind_by` while GitHub reports `BLOCKED`. The practical effect: by the time an agent +reads a check-failure notification, the branch has frequently already moved. + +**Do.** Re-fetch the live `head_sha` and re-read checks and reviews against it before acting. Treat +`FAILURE`, `ERROR`, `CANCELLED`, `TIMED_OUT` and `STARTUP_FAILURE` on a COMPLETED check as failure +(line 343). + +**Do not.** Never hand-merge base into a PR branch merely because `mergeable_state` reads `behind` — +the scheduler owns that mutation. Act only on a check that has actually COMPLETED with a failure +conclusion. + +--- + +## 7. Required checks sit `queued` for hours and never start + +**Symptom.** Nearly every check run on a PR reports `"status": "queued"` with no conclusion, for +hours. This is organization-wide, not per-PR: on 2026-09-05, `.github` had 456 queued runs against 3 +in-progress; `contextual-orchestrator` 93; `noema` 53. + +**Mechanism.** Actions capacity starvation. A queued check is not a failed one: +`scripts/ci/pr_review_merge_scheduler_core.py:342-343` places `QUEUED` in `RUNNING_CHECK_STATES`, +kept separate from `FAILED_CHECK_CONCLUSIONS`. Each `needs:` stage queues separately, so end-to-end +delay routinely exceeds 8 hours, and some runs queued since 2026-08-19 never started at all. There is +no 4-hour upper bound. + +**Do.** Read each check's actual conclusion. Truly `queued` → leave it and work elsewhere. A workflow +that was never assigned a runner → escalate on `.github` #712, #1531, #1219. + +**Never.** Never push an empty commit, close/reopen a PR, or otherwise re-trigger to "kick" CI. A push +invalidates earlier checks and reviews (`AGENTS.md`, "Actions queue and protected-merge procedure"), +and closing cancels in-flight runs (`.github/workflows/noema-review.yml:78`, `cancel-closed-pr-runs`). +Both discard hours of exact-head evidence and re-enter the queue at the back. + +--- + +## Research-grounding freshness KPIs + +Four cheap, repeatable measurements that catch dated-evidence rot before it fails a gate. Measure them +against `origin/main`, never a working tree (see section 0). + +| KPI | Definition | `.github` | `noema` | `contextual-orchestrator` | +|---|---|---|---|---| +| 1. Expired evidence | `valid_until_*` fields whose date is in the past, **excluding** deliberate test fixtures | 0 | 0 | 0 production (4 fixture hits in `tests/test_nim_benchmark_release_acceptance.py`) | +| 2. Silent-aging citations | `as_of*` fields with no paired `valid_until*`, so nothing can ever fail closed | 5 (`scripts/ci/zdr_policy.py:66,90,100,109,117`) | 0 | 0 | +| 3. Open markers | occurrences of `remains open`, `TODO: verify`, `needs-citation`, `unreviewed risk` | 9 | 5 | 42 | +| 4. ADR grounding coverage | fraction of `docs/adr/**` + `docs/planning/adrs/**` files carrying a primary-sources / references section | 3/13 | 2/13 | 29/50 | + +Baseline measured 2026-09-05 against the `main` SHAs listed at the top of this document. + +Two notes that make these numbers honest rather than alarming: + +- KPI 1 is zero in production everywhere. An earlier measurement of this same KPI reported one + expired production item in `contextual-orchestrator`; that reading came from a stale working tree + and was wrong — #1073 had already refreshed the window to `2026-10-05` on main. This is exactly the + failure mode section 0 exists to prevent. +- KPI 2's five hits are the only place in the organization where dated evidence carries no expiry at + all. `contextual_orchestrator/nim_benchmark.py` is currently the org's only implementation of the + `reviewed_at` / `valid_until` pair; adopting the same pair in `scripts/ci/zdr_policy.py`'s + `PROVIDER_ZDR_SCOPE` would give KPI 1 something to measure in `.github` too. + +--- + +## Where the organization's other accumulated know-how lives + +- [`.jules/bolt.md`](../../.jules/bolt.md) — dated performance learnings from prior work on + `scripts/ci/` (regex pre-compilation, `raw_decode` index advancement, N+1 API/subprocess removal), + each as a `**Learning:**` / `**Action:**` pair. +- [`.jules/sentinel.md`](../../.jules/sentinel.md) — dated security learnings from the same surface + (HTML-comment breakout in JSON, cross-language redaction parity, `shell=True` "security theater", + SSRF via redirects), each as `**Vulnerability:**` / `**Learning:**` / `**Prevention:**`. +- `docs/doctoring/` — this directory; per-incident and per-decision records. +- `AGENTS.md` — the binding, tool-agnostic entry point; `CLAUDE.md` complements it for Claude. + +Scan `.jules/sentinel.md` before hardening anything under `scripts/ci/`, and `.jules/bolt.md` before +optimizing it. Those records are why several classes of bug are not re-introduced; they only work if +agents other than the one that wrote them actually read them. From 08c1f8115a9d15327912a4128658930cab9bd500 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 09:14:34 +0000 Subject: [PATCH 02/50] docs(triage): retract the wrong Strix re-run remedy; state why the pin exists Signature 2 told the reader to re-run the failed Strix job, claiming the re-run re-pins trusted_ref to main HEAD. That is false. A reviewer challenged the premise rather than the wording, and checking it against real run data settled it: - workflow_sha is bound when the RUN is created and never re-resolved. Run 33863887675 was created 10:34:35Z and its trusted checkout resolved at 15:35:15Z -- five hours later -- still fetching b15cb994, main's tip at 10:31:52Z. In between main advanced ~20 commits INCLUDING 769691526 (#1851), the Strix fix that would have made the job pass. It failed anyway at 15:50Z against pre-fix source. - A re-run reuses the same run_id, so there is nothing to re-resolve. GitHub documents that a re-run uses the same GITHUB_SHA and GITHUB_REF as the original event. So the retracted advice was not merely imprecise: it would have sent every agent hitting this signature into an unbounded re-run loop against source that cannot change, each attempt consuming a slot in a queue already ~456 deep against ~3 executing -- manufacturing the congestion signature 7 documents. The corrected remedy is to land the fix on main first, then merge current main into the PR head and push, which creates a genuinely new run pinned to a main that carries the fix. Empty commits and close/reopen are named as forbidden rather than left as tempting alternatives. Signature 2 now also states WHY the pin exists before describing its staleness: it is the pull_request_target trust boundary. Without that, the entry reads as if the pin were waste, and the obvious "repair" -- pointing trusted_ref at main -- would convert a supply-chain control into a supply-chain hole in a workflow running with elevated permissions org wide. Signature 3 gains the discriminator the reviewer identified: a re-run is correct for a runtime-external failure (the gateway 502) and useless for a source-staleness failure, and the two look alike. Evidence limits are recorded rather than papered over: no multi-attempt Strix run with logs on both attempts exists in retained history (1,110 runs scanned, one had run_attempt>1 and both attempts were cancelled before the job started), so this rests on creation-time pinning observed directly plus GitHub's documented re-run semantics, not a same-run log diff. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 84 +++++++++++++++---- 1 file changed, 68 insertions(+), 16 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 1c1333f362..74d85692e7 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -16,9 +16,17 @@ in their own records, e.g. [`startup-failure-and-strix-concurrency-20260904.md`](startup-failure-and-strix-concurrency-20260904.md), and [`actions-queue-saturation-hourly-sweep.md`](actions-queue-saturation-hourly-sweep.md). -Baseline for every claim below: `.github` `main@27d7331cc`, `contextual-orchestrator` `main@a080297d`, +Baseline for every claim below: `.github` `main@8aea81323`, `contextual-orchestrator` `main@a080297d`, `noema` `main@e1ac9d5`, all as of 2026-09-05. +**One entry here has already been wrong.** Signature 2 originally told you to re-run the failed job; +a reviewer challenged the premise, and checking it against real run data showed a re-run cannot help, +because `workflow_sha` is bound at run creation. Following the retracted advice would have produced +an unbounded re-run loop feeding the very queue saturation described in signature 7. The entry now +carries the correction and its evidence limits. Treat that as the standard this file is held to: a +plausible mechanism is not a verified one, and an entry that survives a serious attempt to refute it +is worth more than one that reads well. + --- ## 0. Before anything else: your local checkout is probably stale @@ -79,21 +87,56 @@ failure says nothing about it. `ModuleNotFoundError: No module named 'httpx2'`, then `Strix run failed for model 'orchestrator/free' after Ns (exit code 1)`. -**Mechanism.** `.github/workflows/strix.yml:398-400` resolves `trusted_ref` from `job.workflow_sha`, -and line 428 checks the entire trusted Strix source tree out of `ContextualWisdomLab/.github` at that -SHA. That is `.github` main HEAD as of **each run attempt's creation** — not its execution time. A -congestion-queued attempt therefore executes hours later against pre-fix source. The two errors are -different staleness windows: the `AttributeError` means the pin predates `87352d98` (#1783); the -`httpx2` import error means it predates `76969152` (#1851). - -**Do.** Confirm on current main that `scripts/ci/strix_timeout_compat.py` resolves the submodule via -`sys.modules["strix.interface.main"]` and that `requirements-strix-ci-hashes.txt` pins -`httpx2==2.12.0`. Then re-run the failed job: the re-run re-pins to main HEAD and runs the newer -`strix.yml`. - -**Do not.** Do not patch the consuming repository; its head SHA is not implicated. Do not assume the -re-run repeats identical source — a regression landing between your check and the re-run yields a -new, different error, so re-check main and re-run rather than concluding the fix failed. +**Why the pin exists — read this before "repairing" it.** `.github/workflows/strix.yml:398-400` +resolves `trusted_ref` from `job.workflow_sha`, and line 428 checks the entire trusted Strix source +tree out of `ContextualWisdomLab/.github` at exactly that SHA. That pin **is** the +`pull_request_target` trust boundary — it is what stops a pull request from supplying the review +scripts that judge it, and `CLAUDE.md` states the rule directly ("The required review workflows run +the *base branch's* trusted scripts"). The staleness described below is a consequence of that +control, not a defect in it. Do not "fix" it by pointing `trusted_ref` at `main`, at the PR head, or +at any floating ref: that converts a supply-chain control into a supply-chain hole, in a workflow +that runs with elevated permissions against every repository in the organization. The fallback to +the literal string `"main"` at lines 405-409 is dead code in practice — `workflow_sha` is always +populated — and it must stay unreachable. + +**Mechanism.** `workflow_sha` is bound when the **run** is created and is never re-resolved +afterwards. For a `pull_request_target` run it equals `github.sha`, which is the *base branch* tip — +not the PR head, which the run reports separately as `head_sha`. A run that sits queued for hours +therefore executes against base source as of hours ago. Measured on run `33863887675`: created +`10:34:35Z`, the trusted checkout resolved at `15:35:15Z` — five hours later — and fetched +`b15cb994`, which was main's tip at `10:31:52Z`. In the interval main advanced roughly twenty +commits, **including `769691526` (#1851), the very Strix fix that would have made the job pass**; the +job nevertheless failed at `15:50Z` against pre-fix source. The two error strings are two staleness +windows: `AttributeError` means the pin predates `87352d98` (#1783), the `httpx2` import error means +it predates `769691526` (#1851). + +**Do.** First confirm on current main that `scripts/ci/strix_timeout_compat.py` resolves the +submodule via `sys.modules["strix.interface.main"]` and that `requirements-strix-ci-hashes.txt` pins +`httpx2==2.12.0`. Then cause a genuinely **new run**, in that order — the fix must already be on main +*before* the new run is created, because the new run pins itself at *its* creation time. The one +remediation that is both effective and permitted here is to **merge current main into the PR head and +push**: that is a real commit, it makes the branch mergeable anyway, and the resulting +`pull_request_target` run pins to a main that carries the fix. + +**Do not.** **Do not re-run the failed job.** A re-run reuses the same `run_id` and therefore the +same `workflow_sha`; GitHub's own documentation states a re-run "will also use the same `GITHUB_SHA` +(commit SHA) and `GITHUB_REF` (git ref) of the original event", and GitHub staff have confirmed a +re-run "will use the original workflow file". So a re-run re-resolves `trusted_ref` to the identical +stale SHA and reproduces the identical failure — indefinitely, while each attempt consumes a slot in +a queue that is already ~456 deep against ~3 executing (signature 7). This document previously said +the opposite; that instruction was wrong and is retracted. Do not push an empty commit and do not +close and reopen the PR to force a new run either — both are forbidden by the organization's merge +discipline, and the merge-main-in step above already produces the new run legitimately. Do not patch +the consuming repository: its head SHA is not implicated. + +**Evidence limits.** The attempt-to-attempt comparison that would settle this most directly does not +exist in this organization's retained history: across 1,110 scanned run records exactly one Strix run +had `run_attempt > 1` (`33926114577`), and both of its attempts were cancelled before the `strix` job +started, so their logs return HTTP 404. The conclusion above rests on creation-time pinning observed +directly in two real runs (`33863887675`, `33860232589`) plus GitHub's documented re-run semantics — +not on a same-run log diff. A `workflow_dispatch` run should also resolve `workflow_sha` freshly, but +that is inferred from `repository_dispatch` behaviour and the general creation-time rule; no +`workflow_dispatch` Strix run appears in the sample. --- @@ -116,6 +159,15 @@ content-shaped rejections would surface as 400/413 `scripts/ci/pr_review_merge_scheduler_core.py:3746` re-runs Strix only, and `scripts/ci/noema_review_handoff.py`'s dispatch requires a reusable exact-head OpenCode approval. +**Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look +alike — a red required check on a review job — and take opposite actions, so check which one you +have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a +gateway call that failed, so re-executing it issues a fresh call that can succeed. Signature 2 is +*source-staleness*: re-executing pinned source re-executes the same stale source, so a re-run there +is a guaranteed no-op. The discriminator is whether the fix you are waiting on lives on `main` +(signature 2 — you need a new run) or in the transient behaviour of an external service (this +signature — a re-run is exactly right). + **Do not.** Do not blame `served_model`; it is merely the last candidate that failed (`orchestrator.py:7812`). Do not diagnose it as a provider hang specifically — a 502 here equally covers upstream 5xx, TLS, DNS and connection failures. Do not add `timeout-minutes` to From 0ab5fef752b31dd5fc3e2e1eb95d3a3ef3b841a6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 09:19:04 +0000 Subject: [PATCH 03/50] docs(triage): add same-anchor conflict signature and measurement-integrity rules Both contributed by a reviewing session from measurements it made today, not by this session. Recorded here because the catalog is the org's shared store and a lesson only pays off when someone other than its author can find it. Signature 8 -- same-anchor merge conflicts. The collision unit for append-heavy documents is the anchor, not the file. Measured on docs/product-technical-gap-baseline.md: #1868 merged SECOND into the same file and stayed clean because its hunk sat in a different region, while #1903 broke on sharing #1868's anchor. So a path-level claim is wrong in both directions -- it over-serializes PRs that would never have touched and fails to warn the pair that actually conflicts. Also records that `behind` is admin-mergeable with zero pushes while `dirty` is a git-level conflict that cannot be bypassed (405), so the push-free path exists for exactly one PR per anchor per round. Measurement integrity, under the KPI table. `c=$(gh api ... || echo 0)` turns a rate-limited call into a genuine-looking zero; in a real sweep that produced impossible rows (in_progress_runs=13 with running_jobs=0 AND queued_jobs=0) which were briefly explained away as "the metric oscillates" rather than read as the instrument breaking. Re-measured without the mask: min 27 / max 36 / mean 32.1, stable. Three rules follow: never let a failed call fall back to a countable value, treat a self-contradictory row as the tell, and settle a disagreement about a total by printing records rather than recounting. Every KPI in this file is a script-produced count, so these belong beside the table rather than in a separate note. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- AGENTS.md | 7 ++- docs/doctoring/ci-failure-signature-triage.md | 59 ++++++++++++++++++- 2 files changed, 62 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 456809fd1f..9bd0478627 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -191,9 +191,10 @@ they only pay off when an agent other than their author reads them. three repositories (`opencode-review` wait-for-verdict, `strix` stale `workflow_sha`, `noema-review` gateway 502, expired NIM cost evidence, unscoped `pytest` collection, stale `head_sha` notifications, - queued-versus-failed), each with its verified mechanism, the action to take, - and the actions not to take. Also carries the research-grounding freshness - KPIs and their measured baseline. + queued-versus-failed, same-anchor merge conflicts), each with its verified + mechanism, the action to take, and the actions not to take. Also carries the + research-grounding freshness KPIs, their measured baseline, and how to count + them without a masked API error manufacturing a phenomenon. - [`.jules/bolt.md`](.jules/bolt.md) — dated performance learnings for `scripts/ci/` as `**Learning:**` / `**Action:**` pairs. Scan it before optimizing anything on that surface. diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 74d85692e7..99d1739253 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -274,6 +274,42 @@ Both discard hours of exact-head evidence and re-enter the queue at the back. --- +## 8. Your PR goes `dirty` while another PR touching the *same file* merged cleanly + +**Symptom.** Two or more open PRs append to one document. One of them merges without incident; yours +flips to `mergeable_state: dirty` and an admin merge returns **405**. + +**Mechanism.** The collision unit is not the file — it is the **anchor**, the context line git needs +to place a hunk. Measured on `docs/product-technical-gap-baseline.md`: `#1904` appended at +`@@ -3234,0 +3235,119 @@` and `#1868` at `@@ -2775,0 +2776,2 @@`; `#1868` merged **second** into the +same file and stayed clean because its hunk sat in a different region. `#1903` broke — it shares +`#1868`'s anchor, the identical context line `prose" convention already stated in CLAUDE.md.`. So +same-file is not the predictor in either direction: it over-serializes PRs that would never have +touched, and it fails to warn the pair that actually conflicts. + +The two unmergeable states are also not equivalent, and this decides whether you must push: + +| state | meaning | admin merge with `enforce_admins: false` | +|---|---|---| +| `behind` | base advanced; no textual conflict | **succeeds**, zero pushes needed | +| `dirty` | git-level conflict | **cannot be bypassed** — returns 405 | + +So the push-free merge path exists for exactly **one PR per anchor per round**. Everyone after that +needs a real push to resolve, whatever their review state. + +**Do.** When claiming an append-heavy document, claim the *anchor*, not the path — e.g. a lane-claim +marker of the form `paths=docs/.md#`. Ordinary code files can stay +path-granular, since edits there are usually region-local. When you do conflict, resolve by keeping +both sides and then verify nothing was silently dropped: compare `grep -c '^## '` between +`git show origin/main:` and your merge result. A `--ours`/`--theirs` resolution produces zero +conflict markers while deleting an entire section, which reads as a clean merge. + +**Do not.** Do not serialize every PR that touches a shared file — that is the over-correction this +signature exists to prevent, and it stalls work that would have merged fine. Do not assume a clean +merge by a peer means the file is safe for you: they may simply have landed in a different region. + +--- + ## Research-grounding freshness KPIs Four cheap, repeatable measurements that catch dated-evidence rot before it fails a gate. Measure them @@ -297,7 +333,28 @@ Two notes that make these numbers honest rather than alarming: - KPI 2's five hits are the only place in the organization where dated evidence carries no expiry at all. `contextual_orchestrator/nim_benchmark.py` is currently the org's only implementation of the `reviewed_at` / `valid_until` pair; adopting the same pair in `scripts/ci/zdr_policy.py`'s - `PROVIDER_ZDR_SCOPE` would give KPI 1 something to measure in `.github` too. + `PROVIDER_ZDR_SCOPE` gives KPI 1 something to measure in `.github` too. `.github` #1916 does this. + +### Measuring these without manufacturing a phenomenon + +Every KPI above is a count produced by a script, and a counting script fails in a way that looks like +data rather than like an error. + +- **Never let a failed API call fall back to a countable value.** `c=$(gh api ... 2>/dev/null || echo + 0)` turns every rate-limited call into a genuine-looking zero. In a real sweep of this organization + that produced rows reading `in_progress_runs=13` with `running_jobs=0` *and* `queued_jobs=0` — an + impossible combination — and the zeros were initially explained away as "the metric oscillates" + rather than read as the measurement breaking. `gh api rate_limit` afterwards showed the quota had + just reset, confirming it. Re-measured without the mask, 8 samples over 3 minutes across 7 repos + gave min 27 / max 36 / mean 32.1: stable, no oscillation. Fail loudly, or count errors in their own + column. +- **A self-contradictory row is the tell.** Before believing a surprising aggregate, look for a row + that cannot physically exist. That is cheaper than re-deriving the whole measurement and it + distinguishes a broken instrument from a real effect. +- **When two sessions disagree on one number, stop counting and print records.** Aggregates hide both + loop bugs and throttled calls; individual job entries carrying `runner_name` and `started_at` + cannot be forged by either. Disagreement about a total is resolved by listing the underlying rows, + not by re-running the same count more carefully. --- From cf2ed76f651fddb717f4f885955a65ee0620b3ba Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 09:23:18 +0000 Subject: [PATCH 04/50] docs(triage): correct the measurement-defect cause; add CodeQL re-run futility Corrects a wrong causal attribution this file carried for one commit, and adds a third instance of the re-run trap plus the rule that unifies them. The measurement subsection blamed REST rate limiting, citing gh api rate_limit reporting reset_in=3599s. That is a correlate, not a cause: reset_in=3599 only means the hourly window just refreshed, and reads the same whether or not anything was throttled. The real defect, found by direct test after a peer challenged it, is zsh word-splitting -- `for x in $var` does NOT split in zsh while `for x in $(cmd)` does, so `runs=$(gh api ...)` + `for id in $runs` iterates once over the whole blob, produces `invalid control character in URL`, and `|| echo 0` turns that into a zero. bash splits both forms, so the broken version looks portable and is not. The same session's other sweeps used the command-substitution form and were correct all along, which is why one session produced both right and wrong numbers. `|| echo 0` was the concealer, never the cause; that rule stands, its explanation did not. Signature 9 -- CodeQL compat re-run futility. The compat job dispatches a scan and fails fast awaiting a verdict, which is correct only when the dispatched scan can get a runner. On contextual-orchestrator#1032, seven of the last eight CodeQL PR runs were queued and the one completed run was cancelled, the oldest queued 2.5+ hours. So each re-run re-dispatches into the queue that is causing the failure. That makes three signatures where the same red-looking check takes opposite actions, so they now share one stated rule: name the precondition the success depends on and confirm it changed. Transient and runtime-external (3) is re-runnable; pinned stale source (2) and unmet precondition (9) are not. Two entries here once prescribed re-running unconditionally and both were wrong. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- AGENTS.md | 6 +- docs/doctoring/ci-failure-signature-triage.md | 80 +++++++++++++++++-- 2 files changed, 78 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9bd0478627..90dc3cb76e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -191,8 +191,10 @@ they only pay off when an agent other than their author reads them. three repositories (`opencode-review` wait-for-verdict, `strix` stale `workflow_sha`, `noema-review` gateway 502, expired NIM cost evidence, unscoped `pytest` collection, stale `head_sha` notifications, - queued-versus-failed, same-anchor merge conflicts), each with its verified - mechanism, the action to take, and the actions not to take. Also carries the + queued-versus-failed, same-anchor merge conflicts, futile CodeQL compat + re-runs), each with its verified mechanism, the action to take, and the + actions not to take, plus the rule for deciding when a re-run can help at + all. Also carries the research-grounding freshness KPIs, their measured baseline, and how to count them without a masked API error manufacturing a phenomenon. - [`.jules/bolt.md`](.jules/bolt.md) — dated performance learnings for diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 99d1739253..fdfd706dfa 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -310,6 +310,60 @@ merge by a peer means the file is safe for you: they may simply have landed in a --- +## 9. `CodeQL compatibility analysis` fails in seconds, and re-running never helps + +**Symptom.** `CodeQL compatibility analysis (actions)` fails in ~7s. The failing steps are +`Request current-head CodeQL scan dispatch` and +`Release runner or enforce current-head CodeQL verdict`. Re-running produces an identical failure; +the run reaches `run_attempt=2` and beyond with no change. + +**Mechanism.** The compat job is *designed* to dispatch a separate `CodeQL PR` scan and then fail +fast while awaiting its verdict. That is correct behaviour when the dispatched scan can obtain a +runner. Under queue saturation it cannot. Measured on `contextual-orchestrator#1032`: of the last +eight `CodeQL PR` runs in that repository, **seven were `queued` and the one completed run was +`cancelled`**; the oldest queued run had been created at `06:42Z` and was still queued more than two +and a half hours later. Queue depth at the time: 108 runs in `contextual-orchestrator`, 481 in +`.github`, 344 in `naruon`. + +The result is a closed loop: the compat job dispatches → the scan enters a saturated queue and never +starts → no verdict is ever published → the compat job fails fast → a re-run **dispatches again**. +Each re-run cannot succeed *and* enlarges the queue that is causing the failure. + +**Do.** Before re-running anything CodeQL-related, check whether the dispatched scans are actually +starting: + +```bash +gh api "repos///actions/runs?per_page=20" \ + --jq '[.workflow_runs[] | select(.name|test("CodeQL";"i")) | {status,conclusion,created_at}]' +``` + +If those runs are sitting `queued`, treat the compat failure as blocked on runner capacity +(signature 7), not on this PR, and say so on the PR rather than retrying. + +**Do not.** Do not re-run while the dispatched scans are queued: it is pure queue amplification, and +it makes the shared condition worse for every other PR in the organization. Do not read the fast +failure as a defect in the PR's code — the job never got as far as analysing it. + +--- + +## A general rule these three signatures share + +Signatures 2, 3 and 9 all present as a red required check on a review job, and two of the three make +a re-run useless. The discriminator is not the error text but the **precondition**: + +| the failure is… | re-run? | why | +|---|---|---| +| transient and runtime-external (signature 3: gateway 502) | **yes** | a fresh call can succeed; nothing else must change first | +| pinned to stale source (signature 2: `workflow_sha`) | **no** | the same source re-executes; only a *new run* re-pins | +| waiting on a precondition still unmet (signature 9: queued CodeQL scan) | **no** | re-running re-issues the same unmet request and deepens the queue | + +Before re-running any failed check, name the precondition its success depends on and confirm that +precondition has changed. "It might work this time" is not a precondition. Two entries in this +document once prescribed re-running unconditionally; both were wrong, and both were caught by review +rather than by their author. + +--- + ## Research-grounding freshness KPIs Four cheap, repeatable measurements that catch dated-evidence rot before it fails a gate. Measure them @@ -341,13 +395,27 @@ Every KPI above is a count produced by a script, and a counting script fails in data rather than like an error. - **Never let a failed API call fall back to a countable value.** `c=$(gh api ... 2>/dev/null || echo - 0)` turns every rate-limited call into a genuine-looking zero. In a real sweep of this organization - that produced rows reading `in_progress_runs=13` with `running_jobs=0` *and* `queued_jobs=0` — an + 0)` turns *any* failed call into a genuine-looking zero. In a real sweep of this organization that + produced rows reading `in_progress_runs=13` with `running_jobs=0` *and* `queued_jobs=0` — an impossible combination — and the zeros were initially explained away as "the metric oscillates" - rather than read as the measurement breaking. `gh api rate_limit` afterwards showed the quota had - just reset, confirming it. Re-measured without the mask, 8 samples over 3 minutes across 7 repos - gave min 27 / max 36 / mean 32.1: stable, no oscillation. Fail loudly, or count errors in their own - column. + rather than read as the measurement breaking. Re-measured without the mask, 8 samples over 3 + minutes across 7 repos gave min 27 / max 36 / mean 32.1: stable, no oscillation. Fail loudly, or + count errors in their own column. +- **In `zsh`, `for x in $var` does not word-split; `for x in $(cmd)` does.** This was the actual + defect behind the zeros above. `runs=$(gh api ...)` followed by `for id in $runs` iterates **once** + with the whole newline-separated blob as a single word, producing a malformed request + (`invalid control character in URL`) which `|| echo 0` then converted into a zero. The same + session's other measurements used `for id in $(gh api ...)` directly and were correct throughout, + which is exactly why one session produced both right and wrong numbers. `bash` splits both forms, + so the broken version looks portable and is not. Use an explicit reader instead: + `while IFS= read -r id; do ...; done < <(gh api ... --jq '.workflow_runs[].id')`. +- **A correlate is not a cause, even under time pressure.** The zeros above were first attributed to + REST rate limiting on the strength of `gh api rate_limit` reporting `reset_in=3599s`. That number + only means the hourly window had just refreshed — it reads the same whether or not you were ever + throttled. The rate-limit story was published as a cause without being tested, and a peer caught + it; the shell bug above is what a direct test found. An earlier revision of this very section + repeated the rate-limit attribution, which is why it is now spelled out here rather than quietly + replaced. `|| echo 0` was the concealer; it was never the cause. - **A self-contradictory row is the tell.** Before believing a surprising aggregate, look for a row that cannot physically exist. That is cheaper than re-deriving the whole measurement and it distinguishes a broken instrument from a real effect. From 5e849fe4c5677fab37dcb34627384f6a0c679701 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 09:38:27 +0000 Subject: [PATCH 05/50] docs(triage): add secondary rate limit and created_at/updated_at traps Two more measurement traps contributed by a reviewing session, both of which produce confident wrong numbers rather than errors. A full `remaining` does NOT rule out rate limiting. GitHub's secondary (burst) limit returns near-identical 403 text to the primary hourly one while `gh api rate_limit` still reports remaining 5000/5000. Observed: a pulls/1910 fetch 403'd with the budget untouched, the next calls succeeded, and a retry failed again. The remedies are opposite -- primary means wait for reset, secondary means cut concurrency and waiting achieves nothing -- so the misread costs an hour. Read `remaining` first. `created_at` is queue-entry time, `updated_at` is terminal time, and under saturation they are hours apart: one opencode-review run here was created 2026-09-04T10:34:34Z and concluded 2026-09-05T02:12:25Z, a 15.6-hour lifetime. Deriving "how long since this pipeline last concluded a run" from created_at gave ~21 hours where updated_at gives ~7.4 -- same conclusion, number wrong by 3x, and numbers travel further than conclusions. Also query status=success and status=failure explicitly rather than status=completed, since dominant cancellations bury the terminal signal. The print-records rule is restated as the general escape now that three independent mechanisms are known to turn a sweep into a confident zero -- shell word-splitting, error masking, and secondary rate limiting -- none of which announce themselves. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 33 ++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index fdfd706dfa..51585f2c49 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -419,10 +419,35 @@ data rather than like an error. - **A self-contradictory row is the tell.** Before believing a surprising aggregate, look for a row that cannot physically exist. That is cheaper than re-deriving the whole measurement and it distinguishes a broken instrument from a real effect. -- **When two sessions disagree on one number, stop counting and print records.** Aggregates hide both - loop bugs and throttled calls; individual job entries carrying `runner_name` and `started_at` - cannot be forged by either. Disagreement about a total is resolved by listing the underlying rows, - not by re-running the same count more carefully. +- **A full `remaining` does not rule out rate limiting.** GitHub enforces a *secondary* (burst) limit + whose 403 text is near-identical to the primary hourly one, and it fires while + `gh api rate_limit` still reports `remaining 5000 / limit 5000`. Observed here: a + `gh api repos/.../pulls/1910` call returned `403 API rate limit exceeded` with the budget + untouched, calls immediately after it succeeded, and a retry failed again. The remedies are + opposite, so misreading it wastes an hour waiting for a reset that changes nothing: + + | symptom | limit | remedy | + |---|---|---| + | `remaining` at or near `0`, uniform failure | primary (hourly) | wait for `reset` | + | `remaining` full, 403 anyway, intermittent, some calls pass | secondary (burst) | **slow down** — cut concurrency and request rate; waiting does nothing | + + Read `remaining` first: `0` means wait, full-but-still-403 means reduce rate. +- **`created_at` is not `updated_at`, and under queue saturation they are hours apart.** `created_at` + is when a run entered the queue; `updated_at` is when it reached a terminal state. One + `opencode-review` run here was created `2026-09-04T10:34:34Z` and concluded `2026-09-05T02:12:25Z` + — a 15.6-hour lifetime. Asking "how long since this pipeline last produced a terminal run" from + `created_at` reported ~21 hours where `updated_at` gives **~7.4 hours**: the same conclusion, a + number wrong by 3×, and numbers propagate further than conclusions do. Use `created_at` for queue + age, `updated_at` for "when did this last conclude". Relatedly, query `status=success` and + `status=failure` explicitly rather than `status=completed`, because when cancellations dominate + they bury the terminal signal. +- **When two sessions disagree on one number, stop counting and print records.** This is the reliable + escape, because at least three independent mechanisms can each turn a sweep into a confident zero — + shell word-splitting, error masking, and secondary rate limiting — and **none of them announce + themselves**. Aggregates hide all three; individual job entries carrying `runner_name`, + `runner_id` and `started_at` cannot be forged by a loop bug or a throttled API. Disagreement about + a total is resolved by listing the underlying rows, not by re-running the same count more + carefully. --- From aee90b3e217529b3256cbaa85cb92ba3970143bb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 09:42:03 +0000 Subject: [PATCH 06/50] docs(triage): add cancelled-vs-failed and auto-update check starvation Both found by an adversarial re-triage of this session's own open PRs, where the classifier made the first error itself -- which is the evidence that the entry is needed. cancelled is not failure. On #933, 22 check runs on head 9988c4fc were all completed: 20 cancelled, 2 skipped, ZERO failure, with the combined commit status reading success. An automated triage pass still labelled it CI-red and produced a careful analysis of whose fault a failure was that did not exist. Org-wide the cancelled case dominates -- 18 of the 20 most recent agent-review-runtime-quality-ci.yml runs were cancelled -- so reading the conclusion field rather than the surrounding noise is the whole discipline. The starvation loop is the second half. A bot auto-merging main into a branch on a cadence, against the saturated queue of signature 7, means no check can ever conclude: each new head cancels the runs still queued from the last one. On #1722 opencode-agent[bot] merged main in four times across three days; per-head outcomes were cancelled, failure, failure, cancelled, pending, and run 33945594764 sat queued 4.5 hours without ever executing before the next auto-update killed it. That PR has not completed a check cycle once. Consequences recorded: never tell anyone to wait on a specific queued run id, because on an auto-updated branch it may already be cancelled and a cancelled run cannot produce a conclusion; and do not merge main in yourself in response, since a second updater cannot help a branch already updated faster than the queue absorbs. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- AGENTS.md | 3 +- docs/doctoring/ci-failure-signature-triage.md | 37 +++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 90dc3cb76e..7952d3ccb3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -192,7 +192,8 @@ they only pay off when an agent other than their author reads them. `workflow_sha`, `noema-review` gateway 502, expired NIM cost evidence, unscoped `pytest` collection, stale `head_sha` notifications, queued-versus-failed, same-anchor merge conflicts, futile CodeQL compat - re-runs), each with its verified mechanism, the action to take, and the + re-runs, cancelled-versus-failed and auto-update check starvation), each + with its verified mechanism, the action to take, and the actions not to take, plus the rule for deciding when a re-run can help at all. Also carries the research-grounding freshness KPIs, their measured baseline, and how to count diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 51585f2c49..440b6956d7 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -346,6 +346,43 @@ failure as a defect in the PR's code — the job never got as far as analysing i --- +## 10. Every check on the head reads `cancelled`, and the PR never completes a check cycle + +**Symptom.** `get_check_runs` on the current head returns a full set of runs, *all* `status: +"completed"`, with conclusions that are `cancelled` and `skipped` and **no** `failure`. The combined +commit status is `success`. Yet the PR has never once finished a green cycle. + +**`cancelled` is not `failure`, and misreading it manufactures a phantom investigation.** Measured on +`#933`: 22 check runs on head `9988c4fc`, all completed — 20 `cancelled`, 2 `skipped`, **zero +`failure`** — with `get_status` returning `state: "success"`. An automated triage pass over this +repository nonetheless labelled it a CI-red failure and produced a careful, entirely wasted analysis +of whose fault the failure was, for a failure that did not exist. Org-wide this is the common case, +not the exception: 18 of the 20 most recent runs of `agent-review-runtime-quality-ci.yml` were +`cancelled`. Read the *conclusion* field, and treat a head with zero `failure` conclusions as not red +no matter how much red-adjacent noise surrounds it. + +**The starvation loop.** A bot that auto-merges `main` into a branch on a cadence, combined with the +saturated queue of signature 7, prevents any check from ever concluding — each new head cancels the +runs still queued from the previous one. Measured on `#1722`: `opencode-agent[bot]` merged `main` in +at `2026-09-03T05:49:50Z`, `2026-09-03T18:37:16Z`, `2026-09-05T01:32:00Z` and `2026-09-05T09:18:01Z`. +The per-head outcomes were `093c39b0` cancelled, `6dcba3d3` failure, `ad38c487` failure, `e3b0b2d6` +cancelled, `88775b66` pending. Run `33945594764` sat queued for **4.5 hours without ever executing** +and was then killed at `09:18:08Z` by the arrival of the next auto-update. The PR has not completed a +check cycle once in three days. + +**Do.** Establish which head you are on *before* citing any check, and re-establish it after any +delay — on an auto-updated branch the head moves without a human touching it. If the branch is being +auto-updated, the actionable question is not "why did this check fail" but whether the current head +will be allowed to finish before the updater resets it. Escalate the loop itself rather than +triaging its symptoms. + +**Do not.** Do not wait on, or tell anyone else to wait on, a specific queued run id: on an +auto-updated branch that run may already have been cancelled by a newer head, and a cancelled run can +never produce a conclusion. Do not respond by merging `main` in yourself — a second updater does not +help, and the branch is already being updated more often than the queue can absorb. + +--- + ## A general rule these three signatures share Signatures 2, 3 and 9 all present as a red required check on a review job, and two of the three make From d9a4c21dc6e9c243dff273a56bcc0a57bc904abd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 09:46:49 +0000 Subject: [PATCH 07/50] docs(triage): correct the rate-limit mechanism; record the review livelock Consolidated deliberately into one push. Signature 7 now documents that a PR pushed more often than ~5 hours can never pass review, and this branch has been pushed five times in forty minutes -- so this is the last push before letting a run survive. Rate limiting, corrected a second time. The previous revision called it a secondary/burst limit because `gh api rate_limit` reported 5000/5000 during 403s. That endpoint is simply wrong: the same token at the same moment got 403 with X-Ratelimit-Remaining: 0, Used: 5422, Resource: core. It is ordinary primary core exhaustion. Read the failing response's own headers via --include; never diagnose from the rate_limit endpoint. Two consequences follow that the old text got backwards: a 403 on one endpoint says nothing about another because quotas are per-bucket (core at 0 while search 30/30, graphql 5000/5000 and actions/* served 200 in the same 0.5s sweep), and the core quota is shared per USER, so 7+ concurrent sessions re-drained the 5000 refill within a minute of reset -- waiting does not help unless the fleet slows down. This also explains this document's own earlier disputes: silent zero sweeps, mergeable_state reads returning "unknown", PRs appearing to have no checks. Signature 7 gains its root cause: livelock, not slowness. 100 of the 100 most recent completed opencode-review runs were cancelled and the last completed review was 2026-09-04T10:14:59Z; median run lifetime 10.8 minutes against 4.7+ hours needed, 99.9% of it queue wait. Runs die to the next push before ever getting a runner. The concurrency config is already correct and must not be switched to cancel-in-progress: false -- 62% of cancelled jobs never held a runner, so reverting returns all of them to a queue already at 55/60. Resolves a contradiction this file introduced between signatures 6 and 10. Both are right about different questions: FAILED_CHECK_CONCLUSIONS does count CANCELLED, so the scheduler will not merge past it, but cancellation here is overwhelmingly queue sweeping and is never evidence the change is broken. The remedy is a surviving run, not a code fix. Conditions the "main is often the broken side" rule: confirming main's tip is green is necessary but not sufficient, since two internally consistent branches can merge badly wherever the text does not overlap and git marks nothing. Full suite plus exhaustive grep of changed values after any base merge. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 81 ++++++++++++++----- 1 file changed, 63 insertions(+), 18 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 440b6956d7..1400cf9cb2 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -225,6 +225,14 @@ broken main into your branch then loops. Never force-push. Do not treat `tests/test_docs_only_pr_runner_admission.py` as this signature; it is named explicitly by the strix step. +**Confirming main's tip is green is necessary but not sufficient.** Two branches can each be +internally consistent and still produce a broken merge, because git only flags the region where the +text overlaps. `contextual-orchestrator#1068` is the worked counterexample: the evidence and its +validator live in *different* regions of the file, so the merge surfaced a conflict in only one of +them, and resolving that one in isolation left the pair inconsistent — with nothing marked to warn +you. After any base merge, run the **full** suite (not the changed-file subset) and grep exhaustively +for every value the merge changed, including in files git reported as clean. + --- ## 6. A `check_run` failure whose `head_sha` no longer matches the live PR head @@ -242,9 +250,15 @@ checks do **not** stop the second and third paths. `branch_outdated_by_base` (16 the REST compare `behind_by` while GitHub reports `BLOCKED`. The practical effect: by the time an agent reads a check-failure notification, the branch has frequently already moved. -**Do.** Re-fetch the live `head_sha` and re-read checks and reviews against it before acting. Treat -`FAILURE`, `ERROR`, `CANCELLED`, `TIMED_OUT` and `STARTUP_FAILURE` on a COMPLETED check as failure -(line 343). +**Do.** Re-fetch the live `head_sha` and re-read checks and reviews against it before acting. + +**On `CANCELLED`, the scheduler and your triage ask different questions — do not conflate them.** +`FAILED_CHECK_CONCLUSIONS` (line 343) counts `FAILURE`, `ERROR`, `CANCELLED`, `TIMED_OUT` and +`STARTUP_FAILURE`, so a cancelled run **does** block the merge scheduler and you cannot merge past +it. It is not evidence that anything is *wrong with the change*: in this organization cancellation is +overwhelmingly queue sweeping (signature 10). So read it as "this check produced no verdict, and the +scheduler will not merge without one" — never as "this PR is broken". The remedy is a surviving run, +not a code fix. **Do not.** Never hand-merge base into a PR branch merely because `mergeable_state` reads `behind` — the scheduler owns that mutation. Act only on a check that has actually COMPLETED with a failure @@ -262,10 +276,28 @@ in-progress; `contextual-orchestrator` 93; `noema` 53. `scripts/ci/pr_review_merge_scheduler_core.py:342-343` places `QUEUED` in `RUNNING_CHECK_STATES`, kept separate from `FAILED_CHECK_CONCLUSIONS`. Each `needs:` stage queues separately, so end-to-end delay routinely exceeds 8 hours, and some runs queued since 2026-08-19 never started at all. There is -no 4-hour upper bound. +no 4-hour upper bound. The organization-wide ceiling is saturated: 55 of 60 running jobs, with +`.github` holding ~28% of the waiting volume while receiving ~4% of the execution slots. + +**Root cause: livelock, not merely slowness.** Of the 100 most recent completed `opencode-review.yml` +runs, **100 were `cancelled`**; the last review to actually run to completion did so at +`2026-09-04T10:14:59Z`. Median run lifetime is **10.8 minutes** against the **4.7+ hours** a run needs +to reach completion, of which 99.9% is queue wait (confirmed by decomposing per-job `created_at` vs +`started_at`). Runs are cancelled by the next push to the same PR *before they are ever assigned a +runner*. The practical consequence is a hard rule: **a PR pushed more often than roughly every five +hours can never pass review.** If you are iterating on a PR every few minutes, you are not waiting on +the queue — you are resetting it, and no amount of further pushing will produce a verdict. + +**The concurrency configuration is already correct — do not "fix" it.** The group is +`required-opencode-review-{repo}-{PR number}` with `cancel-in-progress: true`, which is right. Do not +switch it to `false`: **62% of the cancelled jobs had not been assigned a runner**, so their +cancellation costs nothing real, and forcing them to run would return all 62% to a queue that is +already at the ceiling. **Do.** Read each check's actual conclusion. Truly `queued` → leave it and work elsewhere. A workflow -that was never assigned a runner → escalate on `.github` #712, #1531, #1219. +that was never assigned a runner → escalate on `.github` #712, #1531, #1219. If a PR of yours has +never completed a review cycle, count your own pushes to it before blaming the queue: batch your +remaining changes into one push and then leave the branch untouched long enough for a run to survive. **Never.** Never push an empty commit, close/reopen a PR, or otherwise re-trigger to "kick" CI. A push invalidates earlier checks and reviews (`AGENTS.md`, "Actions queue and protected-merge procedure"), @@ -456,19 +488,32 @@ data rather than like an error. - **A self-contradictory row is the tell.** Before believing a surprising aggregate, look for a row that cannot physically exist. That is cheaper than re-deriving the whole measurement and it distinguishes a broken instrument from a real effect. -- **A full `remaining` does not rule out rate limiting.** GitHub enforces a *secondary* (burst) limit - whose 403 text is near-identical to the primary hourly one, and it fires while - `gh api rate_limit` still reports `remaining 5000 / limit 5000`. Observed here: a - `gh api repos/.../pulls/1910` call returned `403 API rate limit exceeded` with the budget - untouched, calls immediately after it succeeded, and a retry failed again. The remedies are - opposite, so misreading it wastes an hour waiting for a reset that changes nothing: - - | symptom | limit | remedy | - |---|---|---| - | `remaining` at or near `0`, uniform failure | primary (hourly) | wait for `reset` | - | `remaining` full, 403 anyway, intermittent, some calls pass | secondary (burst) | **slow down** — cut concurrency and request rate; waiting does nothing | - - Read `remaining` first: `0` means wait, full-but-still-403 means reduce rate. +- **Never diagnose rate limiting from `gh api rate_limit` — that endpoint lies.** Measured with the + same token at the same moment: `gh api repos/ContextualWisdomLab/.github/branches/main --include` + returned `403 Forbidden` with `X-Ratelimit-Remaining: 0`, `X-Ratelimit-Used: 5422`, + `X-Ratelimit-Resource: core`, while `gh api rate_limit --jq '.resources.core'` reported + `{remaining: 5000, limit: 5000}`. **Read the failing response's own headers** + (`gh api --include`): `X-RateLimit-Resource` names the exhausted bucket and + `-Remaining` / `-Used` are authoritative. An earlier revision of this section called this a + *secondary/burst* limit on the strength of the `rate_limit` reading; it is ordinary **primary + `core` exhaustion**, and it only looked exotic because the diagnostic everyone reached for was + contradicted by the real responses. +- **A 403 on one endpoint is not evidence about another — check the bucket, not the URL shape.** + Quotas are per-resource. In one 0.5-second sweep `core` sat at `0` (so `pulls/`, `issues/`, + `commits/`, `branches/` and repo metadata all returned 403) while `search` (30/30), `graphql` + (5000/5000) and the Actions endpoints were untouched and returned 200. Not path prefix, not + request speed: bucket. +- **The `core` quota is shared per *user*, not per session, and a fleet of agents will drain it.** + The 403 body names the account (`API rate limit exceeded for user ID …`), so one 5000/hour budget + covers every concurrent session on that account. Observed here: `X-Ratelimit-Used: 5422` before the + window reset and `5000` immediately after, still 403 — the refill was consumed within a minute by + 7+ sessions polling PRs, checks, runs and jobs. Waiting for the reset therefore does **not** help + unless the fleet also slows down. Several of this document's own measurement disputes are + downstream of this: sweeps that returned silent zeros, `mergeable_state` reads that came back + `unknown`, and PRs that appeared to carry no checks at all. +- **Prefer paths that spend no quota.** `git show origin/main:` for file contents, + `raw.githubusercontent.com`, and local `git` for history. Under contention prefer non-`core` + buckets — `search/issues` and `actions/*` stayed available throughout the episode above. - **`created_at` is not `updated_at`, and under queue saturation they are hours apart.** `created_at` is when a run entered the queue; `updated_at` is when it reached a terminal state. One `opencode-review` run here was created `2026-09-04T10:34:34Z` and concluded `2026-09-05T02:12:25Z` From 6c64eff7b72fc8f5890729b921c754c0c72430a4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 11:02:57 +0000 Subject: [PATCH 08/50] docs(triage): correct the merge-tree mechanism; add the content-hash pin case The merge-tree entry stated the wrong mechanism, and my own saved output disproved it. I wrote that the old-form output "reports changed in both without emitting markers". It emits them. The output is diff-formatted, so the line is literally `+<<<<<<< .our`, and the `^` anchor in my grep could not match it: grep -c '^<<<<<<<' -> 0 false negative grep -c '<<<<<<<' -> 2 correct grep -c 'changed in both' -> 2 correct A reviewer challenged the claim and I re-read the file I had saved at diagnosis time; the markers were there the whole time. So the prescription was wrong too: the fix is to drop the `^` anchor, not to abandon the tool. `changed in both` is still preferred as the primary signal because it also covers conflict kinds -- mode changes, rename/rename -- that can produce no content markers, where even the unanchored grep reads clean. The real merge remains the authority wherever a conclusion rides on it. Adds the content-hash pin case found resolving #1187, which is the same family: neither side of the visible conflict is correct. tests/ pins `git hash-object` of workflow files, so when both branches edit the pinned workflow git auto-merges the workflow with no marker and flags only the constant. Ours 20a83d55, main's ade10b37, merged file 2fb3306e -- the correct value derives from a file git never reported as conflicted. Either choice gives zero conflict markers and a failing pin. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 36 +++++++++++++++---- 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 9a674779e7..ec295f6325 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -336,13 +336,37 @@ both sides and then verify nothing was silently dropped: compare `grep -c '^## ' `git show origin/main:` and your merge result. A `--ours`/`--theirs` resolution produces zero conflict markers while deleting an entire section, which reads as a clean merge. -**A cheap conflict pre-check that gives false negatives.** `git merge-tree ` and -grepping its output for `^<<<<<<<` looks like a zero-cost way to ask "will this conflict". It is not: -resolving this document's own conflict against `#1914`, that grep returned **0** while an actual -`git merge --no-commit` produced `CONFLICT (content)` in *both* `AGENTS.md` and `CLAUDE.md`. The -old-form output reports `changed in both` without emitting markers. The authoritative check is a real +**A cheap conflict pre-check whose failure is an anchor bug, not the tool.** `git merge-tree + ` piped to `grep -c '^<<<<<<<'` looks like a zero-cost way to ask "will this conflict", and it +returned **0** while a real `git merge --no-commit` conflicted in both `AGENTS.md` and `CLAUDE.md`. +The reason is not that markers are absent. `merge-tree` emits diff-formatted output, so the marker +line is literally `+<<<<<<< .our` — the `^` anchor simply cannot match it: + +``` +$ git merge-tree "$(git merge-base A B)" A B | grep -c '^<<<<<<<' # 0 — false negative +$ git merge-tree "$(git merge-base A B)" A B | grep -c '<<<<<<<' # 2 — correct +$ git merge-tree "$(git merge-base A B)" A B | grep -c 'changed in both' # 2 — correct +``` + +Prefer `changed in both` as the signal: it also covers conflict kinds (mode changes, rename/rename) +that may produce no content markers at all, where even the unanchored grep would read clean. + +```bash +git merge-tree "$(git merge-base A B)" A B | grep -c 'changed in both' +``` + +This is a pre-filter, not a verdict. Where a conclusion rides on the answer, the authority is a real merge in a scratch worktree — `git worktree add -q --detach /tmp/probe && cd /tmp/probe && -git merge --no-commit --no-ff origin/main` — then `git merge --abort` and remove it. +git merge --no-commit --no-ff origin/main` — then `git merge --abort` and remove it. A zero from the +pre-filter is never evidence that a branch is clean. + +**A conflict where neither side is correct: content-hash pins.** `tests/` carries `git hash-object` +pins of workflow files (`grep -rn 'hash-object' tests/` finds them). When both branches edit the +pinned workflow, git auto-merges *the workflow* with no marker and flags only the constant — so the +correct value is derived from a file git never reported as conflicted, and is neither side's. Measured +on #1187: ours `20a83d55`, main's `ade10b37`, and the merged workflow hashing to `2fb3306e`. Choosing +either side yields a resolution with zero conflict markers that fails the pin assertion. Recompute: +`git hash-object ` after the merge, and write that. **Do not.** Do not serialize every PR that touches a shared file — that is the over-correction this signature exists to prevent, and it stalls work that would have merged fine. Do not assume a clean From 8985e323a0776f9d3c1253b304e05d1f8fe2e381 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 13:20:13 +0000 Subject: [PATCH 09/50] docs(triage): record four verified instances from the same day Batched locally to ride the #1908 rebase in one push rather than four. Signature 1 gains the sub-cause where waiting is useless: the dispatch run exists and is rejected at its first gate. opencode-review-dispatch.yml :126-131 requires actor AND sender to equal the single value of vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR; since #1497 the dispatch goes through the OpenCode App token, so the sender is opencode-agent[bot], and a variable still naming github-actions[bot] kills every run in validate-pr-metadata before any verdict can exist (#1929). The entry previously told the reader to wait for a newer dispatch run; in this case that wait never ends. Mechanism verified from origin/main; the live variable value and run counts are #1929's measurements, cited as such. The "green main tip is necessary but not sufficient" rule gains a second live instance, this one hit directly on contextual-orchestrator#1030: markers at nim_benchmark.py:115-125, the validator enforcing the same URL at :2545 auto-merged with no marker. The three-dot diff showed the branch changing both together; the resolution was checked by comparing the two URLs, not by the absence of markers. Signature 6 gains "re-fetch the live head immediately before you push". A validated fix for #1722 was prepared on head 88775b66 over about fifteen minutes; by push time the auto-updater had moved the head to 375013c5, already carrying the fix. Pushing would have reverted it. git ls-remote origin refs/pull//head is the free check. The measurement section gains "a red baseline is not evidence main is red until you know why". A clean origin/main run reported 2 failed and "main is broken" was nearly posted; the cause was the sandbox lacking declared dependencies, one of them core. Baseline-versus-change in the same environment is the comparison that survives an incomplete environment. Full suite 2883 passed, 1 skipped; coverage 100%; interrogate 100%; git diff --check clean. Doc-only. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index ec295f6325..ffa05dfac7 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -75,6 +75,19 @@ If none is, treat it as a real stall and inspect that run for: receipt-gate reje missing `PR_REVIEW_MERGE_TOKEN` / `OPENCODE_APPROVE_TOKEN` wake credential (`opencode-review-dispatch.yml:7549`), or a verdict published under a non-`opencode-agent` identity. +**A distinct sub-cause where waiting is useless: the dispatch run exists and is rejected at its +first gate.** `.github/workflows/opencode-review-dispatch.yml:126-131` authorizes a +`repository_dispatch` only when *both* `github.triggering_actor` and `github.event.sender.login` +equal the single value of `vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR`. Since #1497 (2026-08-31), +`opencode-review.yml:431` sends that dispatch through the OpenCode GitHub App token, so the sender +is `opencode-agent[bot]`; if the variable still names `github-actions[bot]`, every such run dies in +`validate-pr-metadata` at "Bind workflow inputs to live organization pull request metadata" and no +verdict can ever be published for any head (#1929). Discriminator: open the newest dispatch run for +your head — if its **first** job concluded `failure` with `::error::repository_dispatch +authorization rejected actor=…`, you are in this case, and no amount of waiting or re-running the +required job changes it. Remedy is #1932 (a comma-separated allowlist, mechanism only) plus an owner +updating the variable; neither is a per-PR action. + **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. @@ -233,6 +246,15 @@ them, and resolving that one in isolation left the pair inconsistent — with no you. After any base merge, run the **full** suite (not the changed-file subset) and grep exhaustively for every value the merge changed, including in files git reported as clean. +Second live instance, hit directly on `contextual-orchestrator#1030` the same day: the conflict +markers sat at `nim_benchmark.py:115-125` (the evidence dict, where `main` had refreshed dates on +one citation and the branch had replaced the citation), while the validator that enforces the +citation's URL sat at `:2545` — **2,400 lines away, auto-merged to the branch's value with no +marker**. Taking `main`'s side at the marked hunk would have produced a zero-marker file whose +evidence said `docs/product` and whose validator demanded `run-anywhere`. The tell was the three-dot +diff showing the branch changing evidence and validator *together*; the check that settled it was +grepping both URLs after resolution and comparing them, not the absence of markers. + --- ## 6. A `check_run` failure whose `head_sha` no longer matches the live PR head @@ -252,6 +274,14 @@ reads a check-failure notification, the branch has frequently already moved. **Do.** Re-fetch the live `head_sha` and re-read checks and reviews against it before acting. +**And re-fetch it again immediately before you push, not only before you start.** On 2026-09-05 a +CI failure on #1722 was reproduced on head `88775b66`, fixed by merging `main`, and fully validated — +about fifteen minutes of work. By push time the branch's auto-updater had moved the head to +`375013c5`, which was 0 commits behind `main` and already passed the failing tests. Pushing the +prepared merge would have **reverted the newer commit**. The check is free and takes a second: +`git ls-remote origin refs/pull//head` — if it no longer matches the head you worked from, discard +your merge and re-evaluate on the live head before doing anything else. + **On `CANCELLED`, the scheduler and your triage ask different questions — do not conflate them.** `FAILED_CHECK_CONCLUSIONS` (line 343) counts `FAILURE`, `ERROR`, `CANCELLED`, `TIMED_OUT` and `STARTUP_FAILURE`, so a cancelled run **does** block the merge scheduler and you cannot merge past @@ -594,6 +624,15 @@ data rather than like an error. a total is resolved by listing the underlying rows, not by re-running the same count more carefully. +- **A red baseline is not evidence that `main` is red until you know why it is red.** On 2026-09-05 + a full-suite run of clean `origin/main` for `contextual-orchestrator` reported `2 failed`, and + "main is broken" was a sentence away from being posted org-wide. The cause was the *sandbox*: + `opentelemetry-exporter-otlp-proto-http` — a core dependency in `pyproject.toml`, not an extra — + plus `tiktoken`, `numpy`, `hypothesis` and `fastapi` were simply not installed there. Before + attributing a baseline failure to the repository, list which declared dependencies your environment + is missing (`python -c "import "` per name). The comparison that stays valid in an incomplete + environment is *baseline versus change in the same environment*: an identical failure set plus only + additional passes means the change is clean, whatever the absolute numbers say. --- ## Where the organization's other accumulated know-how lives From 60516ab7e5f44df1f08b17a890d81f574e31c4ab Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 13:42:07 +0000 Subject: [PATCH 10/50] docs(triage): content-hash pin recomputes on every merge, worked twice on #1187 Second conflict on the same constant within four hours: #1932 rewrote the pinned workflow, and because #1187 also edits it the merged file hashed to a third value distinct from both sides. Records that the pin is a function of the merge result, not of recency, and contrasts the live-computed sibling pin that never pays this cost. Batched locally for the #1908 rebase. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index ffa05dfac7..4de778e499 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -398,6 +398,14 @@ on #1187: ours `20a83d55`, main's `ade10b37`, and the merged workflow hashing to either side yields a resolution with zero conflict markers that fails the pin assertion. Recompute: `git hash-object ` after the merge, and write that. +The same constant conflicted a second time on #1187 four hours later, when #1932 landed and rewrote +the pinned workflow: ours `2fb3306e` (the previous recompute), main's `26e85559` (#1932's file), and +the merged file hashing to a **third** value, `0a39def5`, because the branch itself also edits that +workflow and git auto-merged it. So the value is never "whichever side is newer" — it is a function +of the merge result, and it has to be recomputed on *every* merge that touches the pinned file. The +live-computed sibling pin in `test_opencode_rust_coverage_toolchain_contract.py` never pays this +cost; a hardcoded pin pays it on every concurrent change. + **Do not.** Do not serialize every PR that touches a shared file — that is the over-correction this signature exists to prevent, and it stalls work that would have merged fine. Do not assume a clean merge by a peer means the file is safe for you: they may simply have landed in a different region. From 6137f348e3e0613e33cac528cc6f004179b8c31b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 14:20:33 +0000 Subject: [PATCH 11/50] docs(triage): ownership is decided by the PR body, not commit trailers A fingerprint search on noema returned three PRs carrying this session's commit trailer; all three bodies carried the owner's exact-authority section and merge-discipline list, two were driven through a different app, and the one with two of this session's commits was still not this session's to act on. Records the body-not-trailer test in the preamble so it is read before any signature, and ties it to the #933 lane violation it prevents. Batched locally for the #1908 rebase. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 4de778e499..03c541a8e3 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -27,6 +27,19 @@ carries the correction and its evidence limits. Treat that as the standard this plausible mechanism is not a verified one, and an entry that survives a serious attempt to refute it is worth more than one that reads well. +**Before acting on any PR: ownership is decided by the PR body, not by commit trailers.** A branch +can carry your session's `Claude-Session:` trailer on several commits and still not be yours to drive, +because the repository owner may have taken it over since. The test is the body: a section headed +`Current exact authority — KST` (or `Current exact evidence` / `Current authority`) listing +exact head and base SHAs and exact check-run ids, plus an explicit merge-discipline list ("Keep +Draft until …", "Do not … self-approve, force-push …"), means the owner is hand-driving it with +exact-head discipline and no agent edits it, pushes to it, or resolves its threads. Measured +2026-09-05 on `noema`: a fingerprint search returned three "my" PRs (#535, #539, #540); all three +bodies carried that section, two were being driven through a different app entirely, and the one +with two of this session's commits on it was nonetheless not this session's to touch. Searching by +fingerprint finds *history*; only the body tells you *authority*. (#933 earlier the same day was the +failure mode this prevents: an agent merged into a lane the body had explicitly reserved.) + --- ## 0. Before anything else: your local checkout is probably stale From 35a915e6b37fae9cdf91dcbbb556b79733110990 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 14:46:14 +0000 Subject: [PATCH 12/50] docs(triage): occupancy is measured on jobs; the run_id fallback in strix.yml never coalesced push scans Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 03c541a8e3..b1f2a347da 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -337,6 +337,21 @@ switch it to `false`: **62% of the cancelled jobs had not been assigned a runner cancellation costs nothing real, and forcing them to run would return all 62% to a queue that is already at the ceiling. +**What actually holds the slots — measured 2026-09-05T14:27Z.** Occupancy is a property of *jobs*, +not runs: list every in-progress run's jobs (`/actions/runs/{id}/jobs`) and read `started_at` and +`runner_name`. A run "in progress for 10 hours" turned out to be 8 hours of queue plus 2 hours of slot — +`run.created_at` is queue entry, `job.started_at` is slot acquisition. Across the three repositories 18 +jobs held runners; 10 were `strix`, and 5 of those were `push`-on-`main` scans of `.github` commits +already superseded (created 04:09–08:44Z, jobs started 12:31–14:25Z, oldest past 2 hours against a +10–30 minute normal scan), with 4 more `push`/`main` scans queued behind them. Mechanism: `strix.yml`'s +workflow-level concurrency key fell back to `github.run_id` for every non-PR event, so each `main` push +was its own group and no newer `main` head ever retired an older scan — the exact coalescing PR heads +get, missing for the branch that moves most (50 pushes in 24 hours, half within 17 minutes of the +previous). Fix: `.github` #1938 scopes `push` events as `push-`. The general lesson: a +`run_id` fallback in a concurrency key is "never coalesce", and it is safe only for events that +genuinely cannot supersede one another (`schedule`, PR-less dispatch); check every event class the +workflow accepts before accepting that fallback. + **Do.** Read each check's actual conclusion. Truly `queued` → leave it and work elsewhere. A workflow that was never assigned a runner → escalate on `.github` #712, #1531, #1219. If a PR of yours has never completed a review cycle, count your own pushes to it before blaming the queue: batch your From cc25c83571b340c48da79cebe01e8dc2b62452a2 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 16:17:37 +0000 Subject: [PATCH 13/50] docs(triage): a draft PR is invisible to every gate; 33/138, 16/16 and 17/64 open PRs were drafts Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- AGENTS.md | 3 ++- docs/doctoring/ci-failure-signature-triage.md | 27 +++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 566a98c26e..407c74c696 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -236,7 +236,8 @@ they only pay off when an agent other than their author reads them. `workflow_sha`, `noema-review` gateway 502, expired NIM cost evidence, unscoped `pytest` collection, stale `head_sha` notifications, queued-versus-failed, same-anchor merge conflicts, futile CodeQL compat - re-runs, cancelled-versus-failed and auto-update check starvation), each + re-runs, cancelled-versus-failed and auto-update check starvation, and + drafts that no gate will ever approve, review, or rebase), each with its verified mechanism, the action to take, and the actions not to take, plus the rule for deciding when a re-run can help at all. Also carries the diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index b1f2a347da..41ff55ae03 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -513,6 +513,33 @@ help, and the branch is already being updated more often than the queue can abso --- +## 11. Checks are green, no review ever appears, and the scheduler never picks the PR up — it is a draft + +**Symptom.** Every check on the head completes, CodeRabbit posts "Draft PR not reviewed", no +`opencode-agent` or `cwl-noema-review` review ever appears, the auto-rebase never touches the branch, +and the PR is never named in a scheduler candidate list. Nothing on the PR says why. + +**Mechanism.** Four independent gates treat a draft as out of scope, and none of them reports it on +the PR: `scripts/ci/opencode_review_receipt_gate.py` ("draft must never receive bot APPROVE" — an +approval on a draft is not a formal receipt), `scripts/ci/noema_review_gate.py` ("PR is draft; Noema +review skipped"), `scripts/ci/pr_auto_rebase.py` (`"draft PR"` disqualifies the candidate), and +CodeRabbit's default `auto_review.drafts: false`. Every agent harness in this fleet creates pull +requests as drafts by default, so the mismatch is systemic, not a one-off: measured 2026-09-05T16:20Z, +33 of 138 open `.github` PRs, **16 of 16** open `noema` PRs, and 17 of 64 open `contextual-orchestrator` +PRs were drafts — each un-approvable until someone flips it, however green its checks. + +**Do.** Convert your own PR to ready-for-review the moment its local gates pass. The MCP +`update_pull_request` tool takes `draft: false`; REST `PATCH /pulls/{n}` cannot change draft state +(it needs GraphQL `markPullRequestReadyForReview`, which the MCP tool wraps). The `ready_for_review` +event re-fans the required workflows on the same head, so flip before the head accumulates checks you +would rather keep, and never in the middle of a push you are still batching. + +**Never.** Never flip a PR you did not open — a draft may be deliberate work in progress, and the +ownership rule is the same as everywhere else in this catalog (the PR body decides). Send the list to +its owners instead. + +--- + ## Absence of data flow is not evidence that an edge is safe to cut Under queue saturation every serial `needs:` hop costs a full queue round (measured at 1.89–2.76 h From 6c39ff80c120ae7ac497d12dcea4f6d0c04e93b6 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 16:20:38 +0000 Subject: [PATCH 14/50] docs(triage): placeholder started_at on unallocated jobs; a cancelled run may already have acted Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 41ff55ae03..8ab0926f87 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -340,7 +340,10 @@ already at the ceiling. **What actually holds the slots — measured 2026-09-05T14:27Z.** Occupancy is a property of *jobs*, not runs: list every in-progress run's jobs (`/actions/runs/{id}/jobs`) and read `started_at` and `runner_name`. A run "in progress for 10 hours" turned out to be 8 hours of queue plus 2 hours of slot — -`run.created_at` is queue entry, `job.started_at` is slot acquisition. Across the three repositories 18 +`run.created_at` is queue entry, `job.started_at` is slot acquisition. Two discriminators that hold up (peer-verified the same day): a job that never +got a runner reports a placeholder `started_at` equal to the run's `created_at`, so +`completed_at − started_at` on such a job measures queue time, not execution; the reliable tests +are `steps > 0` and `created_at < started_at`. Across the three repositories 18 jobs held runners; 10 were `strix`, and 5 of those were `push`-on-`main` scans of `.github` commits already superseded (created 04:09–08:44Z, jobs started 12:31–14:25Z, oldest past 2 hours against a 10–30 minute normal scan), with 4 more `push`/`main` scans queued behind them. Mechanism: `strix.yml`'s @@ -511,6 +514,12 @@ auto-updated branch that run may already have been cancelled by a newer head, an never produce a conclusion. Do not respond by merging `main` in yourself — a second updater does not help, and the branch is already being updated more often than the queue can absorb. +**The converse trap.** `conclusion == "success"` is not a safe filter for "did this run act". While +chasing an unexplained branch update on 2026-09-05, a peer found a run that had acquired a runner and +pushed an update-branch three seconds before the cancel reached it; it reports `cancelled`, and a +filter on `success` hides it. Attribute side effects (pushes, comments, statuses) by looking for the +side effect itself — `steps > 0`, the commit's own pusher, the comment's author — never by conclusion. + --- ## 11. Checks are green, no review ever appears, and the scheduler never picks the PR up — it is a draft From 5932e36a9916547acefde4ce3d4f37b3c0849609 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 16:24:10 +0000 Subject: [PATCH 15/50] docs(triage): noema 502 rate measured at 50% with a single-upstream free pool; re-run is a coin flip mid-stall Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 8ab0926f87..02854e25a4 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -185,6 +185,16 @@ content-shaped rejections would surface as 400/413 `scripts/ci/pr_review_merge_scheduler_core.py:3746` re-runs Strix only, and `scripts/ci/noema_review_handoff.py`'s dispatch requires a reusable exact-head OpenCode approval. +**Measured 2026-09-05 (10:00–16:25Z), which qualifies the "re-run" above.** Of 14 completed, +non-cancelled `noema-review` runs in `.github`, 7 succeeded and 7 failed, six of them this 502 after +180–2174 s of held runner (about 57 minutes of slot for zero verdicts). The policy report of a failing +job listed 21 free-pool candidates, all `nvidia_nim` / `nvidia_nim_sub` — one upstream — so the +failover loop cannot leave a stalled upstream whatever its retry budget (root cause and design +directions: `.github` #1903). While the stall is measurably ongoing (failure rate near 50% over the +last hour), a re-run is a coin flip that costs another 3–36 minutes of a slot the queue is starving +for. Measure before re-running: list `noema-review.yml` runs from the last hour and grep the failed +jobs' logs for `HTTP Error 502`; re-run once the rate has dropped, not while it is high. + **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a From a50eeb2b3b3e9402b10510005e49447d607c6ce9 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 16:25:40 +0000 Subject: [PATCH 16/50] docs(triage): owner holds override the draft-flip rule; put an owner-session marker on every agent PR Counter-finding from #1912: the owner reverted a peer's flip on noema#552 and holds contextual-orchestrator#1070 / noema#553 as drafts explicitly. Every session shares one login, so ownership goes in the PR body's first line as the lane-claim/owner-session marker already used on #1938. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 02854e25a4..1e9cb3f042 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -555,7 +555,21 @@ would rather keep, and never in the middle of a push you are still batching. **Never.** Never flip a PR you did not open — a draft may be deliberate work in progress, and the ownership rule is the same as everywhere else in this catalog (the PR body decides). Send the list to -its owners instead. +its owners instead. And never flip a draft that carries an owner hold, even one you opened: the +repository owner converted `noema#552` back to draft 27 minutes after a peer flipped it, with no +comment, on an idle PR with clean checks; `contextual-orchestrator#1070` carries an owner comment +"Left as Draft per your instructions — no self-approval, no ready-for-review flip", and `noema#553`'s +body says "Keep Draft until this unchanged exact head receives current terminal CI …" (all three +recorded by `.github` #1912). Before flipping, grep the body and the comment thread for +"keep draft", "ready-for-review", "self-approve": a standing per-PR hold overrides this signature. + +**Ownership marker.** Every session on this account commits and opens PRs as the same login, so +`user.login` attributes nothing and a session can only flip drafts it has direct memory of opening. +Put the owner on the PR body's first line when you open it, in the form already used by `.github` +#1938: ``. It is full-text searchable +(`"owner-session=" in:body`), it survives squash-merges as PR metadata, and it turns "your own PR" +from a memory test into a grep: flip only a draft whose marker is yours and whose thread carries no +hold. --- From c0486784acb3110a2852f58209c6086732547261 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 16:40:33 +0000 Subject: [PATCH 17/50] docs(triage): record the by-hand relief for superseded push/main Strix scans and its three-part test Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 1e9cb3f042..bea02915ed 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -365,6 +365,19 @@ previous). Fix: `.github` #1938 scopes `push` events as `push-`. The g genuinely cannot supersede one another (`schedule`, PR-less dispatch); check every event class the workflow accepts before accepting that fallback. +**Relief you may apply by hand, and the test for it (applied 2026-09-05T16:36Z).** With zero merges +org-wide for 2 h 18 m and six of thirty running `.github` jobs still superseded `push`/`main` Strix +scans, the seven superseded runs were retired and the tip's own scan kept; all seven read +`completed/cancelled` within 20 seconds and six runners returned to the pool. The test has three +parts and all three must hold: the head is an ancestor of the kept tip (`git merge-base +--is-ancestor origin/main`), nothing consumes the run (a push scan covers the whole tree and +publishes no `strix` status), and the slot has been held longer than a normal scan takes. The +workflow comment names "an explicit operator action or a superseded head" as the two legitimate +reasons to retire a run, so this is inside policy, not around it. Use the MCP `actions_run_trigger` +tool's `cancel_workflow_run` method and verify each run with `actions_get` `get_workflow_run`. +Never apply this to PR scans — a PR scan is consumed by its check, and the PR-scoped group already +retires superseded heads on its own. + **Do.** Read each check's actual conclusion. Truly `queued` → leave it and work elsewhere. A workflow that was never assigned a runner → escalate on `.github` #712, #1531, #1219. If a PR of yours has never completed a review cycle, count your own pushes to it before blaming the queue: batch your From de3519db22dd51e773f0933bd137876d3039e1c1 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 17:01:16 +0000 Subject: [PATCH 18/50] docs(triage): the single-upstream pool is the launcher's evidence_only stripping; #1476 is the unblocker Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index bea02915ed..0a5884de06 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -188,9 +188,12 @@ content-shaped rejections would surface as 400/413 **Measured 2026-09-05 (10:00–16:25Z), which qualifies the "re-run" above.** Of 14 completed, non-cancelled `noema-review` runs in `.github`, 7 succeeded and 7 failed, six of them this 502 after 180–2174 s of held runner (about 57 minutes of slot for zero verdicts). The policy report of a failing -job listed 21 free-pool candidates, all `nvidia_nim` / `nvidia_nim_sub` — one upstream — so the +job listed 12 probed routes, all `nvidia_nim` / `nvidia_nim_sub` — one upstream — so the failover loop cannot leave a stalled upstream whatever its retry budget (root cause and design -directions: `.github` #1903). While the stall is measurably ongoing (failure rate near 50% over the +directions: `.github` #1903). The policy layer had admitted 62 free-pool routes across three +accounts; the gap is `scripts/ci/contextual_orchestrator_review_launcher.py`'s +`_routable_discovered_models()` dropping every OpenRouter row as `evidence_only` before serving, +which `.github` #1476 fixes — that PR is the unblocker for this whole signature, not a re-run. While the stall is measurably ongoing (failure rate near 50% over the last hour), a re-run is a coin flip that costs another 3–36 minutes of a slot the queue is starving for. Measure before re-running: list `noema-review.yml` runs from the last hour and grep the failed jobs' logs for `HTTP Error 502`; re-run once the rate has dropped, not while it is high. From 2795b5ecdede78cbdeaf489c088b28ea03b9da32 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 17:05:51 +0000 Subject: [PATCH 19/50] =?UTF-8?q?docs(triage):=20count=20required=20contex?= =?UTF-8?q?ts,=20not=20check-runs=20=E2=80=94=20a=20head=20that=20read=20"?= =?UTF-8?q?22=20green"=20was=207/12?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 0a5884de06..f84534804c 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -665,6 +665,15 @@ Two notes that make these numbers honest rather than alarming: ### Measuring these without manufacturing a phenomenon +**Count required contexts, not check-runs (peer-corrected 2026-09-05).** "22 checks green" on a head +counted all 34 check-runs, most of them non-required; branch protection evaluates the latest +check-run per *required context* (12 on `.github` `main`), and on those the same head was 7/12 with +two designed-pending CodeQL failures and three queued — the state of every non-draft PR that day +(0 of 105 with a SUCCESS rollup at 16:52Z). Read the required list from +`/branches/main/protection/required_status_checks` (or the merge scheduler's rollup) and take the +latest run per context name; a check-run tally mixes required with informational and old with +current, and overstates readiness every time. + Every KPI above is a count produced by a script, and a counting script fails in a way that looks like data rather than like an error. From df2979fca217a0cfed5cbfb152c79e54682ede24 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 17:07:46 +0000 Subject: [PATCH 20/50] docs(triage): #1476 and #1382's policy-layer evidence-only filter are mutually exclusive designs Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index f84534804c..7464c79fee 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -193,7 +193,7 @@ failover loop cannot leave a stalled upstream whatever its retry budget (root ca directions: `.github` #1903). The policy layer had admitted 62 free-pool routes across three accounts; the gap is `scripts/ci/contextual_orchestrator_review_launcher.py`'s `_routable_discovered_models()` dropping every OpenRouter row as `evidence_only` before serving, -which `.github` #1476 fixes — that PR is the unblocker for this whole signature, not a re-run. While the stall is measurably ongoing (failure rate near 50% over the +which `.github` #1476 fixes — that PR is the unblocker for this whole signature, not a re-run. #1382's original design added the opposite filter in the policy layer (`EVIDENCE_ONLY_PROVIDERS = {"openrouter"}`); it was dropped in that PR's conflict resolution in favour of `main`'s `FREE_POOL_CREDENTIAL_NAMES`, which names `OPENROUTER_API_KEY` as a free-pool contributor. The two designs are mutually exclusive: do not reintroduce a policy-layer evidence-only strip, and do not remove OpenRouter from `FREE_POOL_CREDENTIAL_NAMES`, without the routing-policy owner deciding — either re-creates the single-upstream pool. While the stall is measurably ongoing (failure rate near 50% over the last hour), a re-run is a coin flip that costs another 3–36 minutes of a slot the queue is starving for. Measure before re-running: list `noema-review.yml` runs from the last hour and grep the failed jobs' logs for `HTTP Error 502`; re-run once the rate has dropped, not while it is high. From 6aeadc59dace7ea672e0dc2f75008c809aad0b9b Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 17:14:58 +0000 Subject: [PATCH 21/50] =?UTF-8?q?docs(triage):=20correct=20the=20single-up?= =?UTF-8?q?stream=20mechanism=20=E2=80=94=20the=20sidecar's=20account-cap?= =?UTF-8?q?=20default=20of=208,=20not=20evidence=5Fonly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 7464c79fee..6a8abd3d7b 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -190,10 +190,7 @@ non-cancelled `noema-review` runs in `.github`, 7 succeeded and 7 failed, six of 180–2174 s of held runner (about 57 minutes of slot for zero verdicts). The policy report of a failing job listed 12 probed routes, all `nvidia_nim` / `nvidia_nim_sub` — one upstream — so the failover loop cannot leave a stalled upstream whatever its retry budget (root cause and design -directions: `.github` #1903). The policy layer had admitted 62 free-pool routes across three -accounts; the gap is `scripts/ci/contextual_orchestrator_review_launcher.py`'s -`_routable_discovered_models()` dropping every OpenRouter row as `evidence_only` before serving, -which `.github` #1476 fixes — that PR is the unblocker for this whole signature, not a re-run. #1382's original design added the opposite filter in the policy layer (`EVIDENCE_ONLY_PROVIDERS = {"openrouter"}`); it was dropped in that PR's conflict resolution in favour of `main`'s `FREE_POOL_CREDENTIAL_NAMES`, which names `OPENROUTER_API_KEY` as a free-pool contributor. The two designs are mutually exclusive: do not reintroduce a policy-layer evidence-only strip, and do not remove OpenRouter from `FREE_POOL_CREDENTIAL_NAMES`, without the routing-policy owner deciding — either re-creates the single-upstream pool. While the stall is measurably ongoing (failure rate near 50% over the +directions: `.github` #1903). The policy layer had admitted 62 free-pool routes across three accounts; the gap is the sidecar's own cap override — `scripts/ci/contextual_orchestrator_review_sidecar.sh:43` exports `ORCHESTRATOR_CATALOG_ACCOUNT_CAP` with a default of **8** while `policy.DEFAULT_ACCOUNT_CAP` is 4, and `build_zdr_prioritized_catalog` fills its 12-route limit in `(cost, zdr, provider, model)` order, so alphabetically `nvidia_nim` takes 8, `nvidia_nim_sub` takes 4, and `openrouter` is never reached (peer-refuted 2026-09-05: an earlier version of this entry blamed the launcher's `evidence_only` strip, which does not fire on the current pin — CO#949 is vendored). `.github` #1476 is the hardening for a regressed pin, not today's lever; the lever is the cap default and a fill that round-robins across accounts within a tier. Same family as #1415 and #1921: an alphabetical tiebreak plus a budget starves whatever sorts last. #1382's original policy-layer `EVIDENCE_ONLY_PROVIDERS = {"openrouter"}` filter would recreate the single-upstream pool by another route; do not reintroduce it, and do not remove OpenRouter from `FREE_POOL_CREDENTIAL_NAMES`, without the routing-policy owner deciding. While the stall is measurably ongoing (failure rate near 50% over the last hour), a re-run is a coin flip that costs another 3–36 minutes of a slot the queue is starving for. Measure before re-running: list `noema-review.yml` runs from the last hour and grep the failed jobs' logs for `HTTP Error 502`; re-run once the rate has dropped, not while it is high. From e0001c2e8100eab238eef25ede1bd0845c1f4099 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 17:21:50 +0000 Subject: [PATCH 22/50] docs(triage): the selection-order lever is #1939 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 6a8abd3d7b..150bd99f70 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -190,7 +190,7 @@ non-cancelled `noema-review` runs in `.github`, 7 succeeded and 7 failed, six of 180–2174 s of held runner (about 57 minutes of slot for zero verdicts). The policy report of a failing job listed 12 probed routes, all `nvidia_nim` / `nvidia_nim_sub` — one upstream — so the failover loop cannot leave a stalled upstream whatever its retry budget (root cause and design -directions: `.github` #1903). The policy layer had admitted 62 free-pool routes across three accounts; the gap is the sidecar's own cap override — `scripts/ci/contextual_orchestrator_review_sidecar.sh:43` exports `ORCHESTRATOR_CATALOG_ACCOUNT_CAP` with a default of **8** while `policy.DEFAULT_ACCOUNT_CAP` is 4, and `build_zdr_prioritized_catalog` fills its 12-route limit in `(cost, zdr, provider, model)` order, so alphabetically `nvidia_nim` takes 8, `nvidia_nim_sub` takes 4, and `openrouter` is never reached (peer-refuted 2026-09-05: an earlier version of this entry blamed the launcher's `evidence_only` strip, which does not fire on the current pin — CO#949 is vendored). `.github` #1476 is the hardening for a regressed pin, not today's lever; the lever is the cap default and a fill that round-robins across accounts within a tier. Same family as #1415 and #1921: an alphabetical tiebreak plus a budget starves whatever sorts last. #1382's original policy-layer `EVIDENCE_ONLY_PROVIDERS = {"openrouter"}` filter would recreate the single-upstream pool by another route; do not reintroduce it, and do not remove OpenRouter from `FREE_POOL_CREDENTIAL_NAMES`, without the routing-policy owner deciding. While the stall is measurably ongoing (failure rate near 50% over the +directions: `.github` #1903). The policy layer had admitted 62 free-pool routes across three accounts; the gap is the sidecar's own cap override — `scripts/ci/contextual_orchestrator_review_sidecar.sh:43` exports `ORCHESTRATOR_CATALOG_ACCOUNT_CAP` with a default of **8** while `policy.DEFAULT_ACCOUNT_CAP` is 4, and `build_zdr_prioritized_catalog` fills its 12-route limit in `(cost, zdr, provider, model)` order, so alphabetically `nvidia_nim` takes 8, `nvidia_nim_sub` takes 4, and `openrouter` is never reached (peer-refuted 2026-09-05: an earlier version of this entry blamed the launcher's `evidence_only` strip, which does not fire on the current pin — CO#949 is vendored). `.github` #1476 is the hardening for a regressed pin, not today's lever; the lever is a fill that round-robins across accounts within a tier — `.github` #1939 (host 1 session), which keeps the sidecar's cap default and argues the cap no longer decides diversity once the fill interleaves. Same family as #1415 and #1921: an alphabetical tiebreak plus a budget starves whatever sorts last. #1382's original policy-layer `EVIDENCE_ONLY_PROVIDERS = {"openrouter"}` filter would recreate the single-upstream pool by another route; do not reintroduce it, and do not remove OpenRouter from `FREE_POOL_CREDENTIAL_NAMES`, without the routing-policy owner deciding. While the stall is measurably ongoing (failure rate near 50% over the last hour), a re-run is a coin flip that costs another 3–36 minutes of a slot the queue is starving for. Measure before re-running: list `noema-review.yml` runs from the last hour and grep the failed jobs' logs for `HTTP Error 502`; re-run once the rate has dropped, not while it is high. From d3dccef221262230ccbf104f5edffe21d5559b09 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 21:23:54 +0000 Subject: [PATCH 23/50] docs(triage): #1939 closes the stall; stalled heads recover by a base-merge push, not a re-run Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 150bd99f70..0279c50848 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -195,6 +195,14 @@ last hour), a re-run is a coin flip that costs another 3–36 minutes of a slot for. Measure before re-running: list `noema-review.yml` runs from the last hour and grep the failed jobs' logs for `HTTP Error 502`; re-run once the rate has dropped, not while it is high. +**Closed 2026-09-05T17:25Z: #1939 (round-robin catalog fill) is on `main`, owner-merged.** A head +whose Strix or Noema run failed on the stall before that does not recover by re-run: `workflow_sha` +is bound at run creation, so the re-run executes the pre-#1939 sidecar (signature 2's mechanism, now +on this signature's side of the line). The remedy for such a head is one push that merges `main` — +a new event binds the current sidecar — and it is worth doing even mid-batch, because the review that +would have been reset had already failed. Applied to four heads at 21:15–21:21Z, each after the full +local gate. + **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a From e9a4f102caf5040f6db36c30896037201479cd42 Mon Sep 17 00:00:00 2001 From: seonghobae Date: Sat, 5 Sep 2026 21:29:56 +0000 Subject: [PATCH 24/50] docs(triage): address CodeRabbit round on the catalog (pacing rule, merge verification, marker trust) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Signature 3: the five-hour pacing figure is a measurement under the 2026-09-04 queue depth and runner ceiling, not an upper bound; the rule is probabilistic ("usually fails to reach a verdict"), not "never". - Signature 6: verify a conflict resolution by diffing the exact `##` heading lists and reading the complete merge diff; a heading count is blind to a deleted-plus-duplicated section and to losses inside one. - Signature 6: tag the merge-tree example as bash and drop the `$` prompts (MD040/MD014). - Signature 11: the owner-session marker is supporting information only — any body editor can paste one — so a draft is flipped only with an independent record that this session created the PR; escape the line-leading `#1938` reference (MD018). - Measurement traps: an identical failure set in an incomplete environment means "no additional observed failures", not "clean"; required gates still run in a supported environment. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 47 ++++++++++++------- 1 file changed, 31 insertions(+), 16 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 0279c50848..1e6b9f838d 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -345,9 +345,12 @@ runs, **100 were `cancelled`**; the last review to actually run to completion di `2026-09-04T10:14:59Z`. Median run lifetime is **10.8 minutes** against the **4.7+ hours** a run needs to reach completion, of which 99.9% is queue wait (confirmed by decomposing per-job `created_at` vs `started_at`). Runs are cancelled by the next push to the same PR *before they are ever assigned a -runner*. The practical consequence is a hard rule: **a PR pushed more often than roughly every five -hours can never pass review.** If you are iterating on a PR every few minutes, you are not waiting on -the queue — you are resetting it, and no amount of further pushing will produce a verdict. +runner*. The practical consequence is a pacing rule, not a law: **a PR pushed more often than roughly +every five hours will usually fail to reach a verdict.** The 4.7-hour figure is what completion took +under the queue depth and the 60-job runner ceiling measured in §7 on 2026-09-04; it is not an upper +bound, and a shallower queue produces verdicts sooner. If you are iterating on a PR every few minutes, +you are not waiting on the queue — you are resetting it, and each further push sends the run to the +back of it. **The concurrency configuration is already correct — do not "fix" it.** The group is `required-opencode-review-{repo}-{PR number}` with `cancel-in-progress: true`, which is right. Do not @@ -424,9 +427,12 @@ needs a real push to resolve, whatever their review state. **Do.** When claiming an append-heavy document, claim the *anchor*, not the path — e.g. a lane-claim marker of the form `paths=docs/.md#`. Ordinary code files can stay path-granular, since edits there are usually region-local. When you do conflict, resolve by keeping -both sides and then verify nothing was silently dropped: compare `grep -c '^## '` between -`git show origin/main:` and your merge result. A `--ours`/`--theirs` resolution produces zero -conflict markers while deleting an entire section, which reads as a clean merge. +both sides and then verify nothing was silently dropped: diff the exact `##` heading *lists* of +`git show origin/main:` and of your merge result (a count is blind to one section deleted and +another duplicated), then read the complete merge diff before accepting the resolution — altered, +duplicated, or lost content *inside* a section leaves every heading in place. A `--ours`/`--theirs` +resolution produces zero conflict markers while deleting an entire section, which reads as a clean +merge. **A cheap conflict pre-check whose failure is an anchor bug, not the tool.** `git merge-tree ` piped to `grep -c '^<<<<<<<'` looks like a zero-cost way to ask "will this conflict", and it @@ -434,10 +440,10 @@ returned **0** while a real `git merge --no-commit` conflicted in both `AGENTS.m The reason is not that markers are absent. `merge-tree` emits diff-formatted output, so the marker line is literally `+<<<<<<< .our` — the `^` anchor simply cannot match it: -``` -$ git merge-tree "$(git merge-base A B)" A B | grep -c '^<<<<<<<' # 0 — false negative -$ git merge-tree "$(git merge-base A B)" A B | grep -c '<<<<<<<' # 2 — correct -$ git merge-tree "$(git merge-base A B)" A B | grep -c 'changed in both' # 2 — correct +```bash +git merge-tree "$(git merge-base A B)" A B | grep -c '^<<<<<<<' # prints 0 — false negative +git merge-tree "$(git merge-base A B)" A B | grep -c '<<<<<<<' # prints 2 — correct +git merge-tree "$(git merge-base A B)" A B | grep -c 'changed in both' # prints 2 — correct ``` Prefer `changed in both` as the signal: it also covers conflict kinds (mode changes, rename/rename) @@ -586,11 +592,16 @@ recorded by `.github` #1912). Before flipping, grep the body and the comment thr **Ownership marker.** Every session on this account commits and opens PRs as the same login, so `user.login` attributes nothing and a session can only flip drafts it has direct memory of opening. -Put the owner on the PR body's first line when you open it, in the form already used by `.github` -#1938: ``. It is full-text searchable -(`"owner-session=" in:body`), it survives squash-merges as PR metadata, and it turns "your own PR" -from a memory test into a grep: flip only a draft whose marker is yours and whose thread carries no -hold. +Put the owner on the PR body's first line when you open it, in the form `.github` PR `#1938` already +uses: ``. It is full-text searchable +(`"owner-session=" in:body`) and it survives squash-merges as PR metadata, so it makes "which of the +open drafts might be mine" a grep instead of a memory test. It is *not* authorization on its own: +anyone who can edit a PR body can paste a marker, so a marker that names your session is a hint to +go and check, never proof. Flip a draft only when all three hold — the marker names your session, +your own record shows you created that PR (the `create_pull_request` result carrying its number in +your transcript, or the PR appearing in `list_pull_requests` for a head branch you pushed), and the +body and thread carry no hold. A marker without that independent record is treated exactly like a +missing one: the PR belongs to someone else. --- @@ -758,7 +769,11 @@ data rather than like an error. attributing a baseline failure to the repository, list which declared dependencies your environment is missing (`python -c "import "` per name). The comparison that stays valid in an incomplete environment is *baseline versus change in the same environment*: an identical failure set plus only - additional passes means the change is clean, whatever the absolute numbers say. + additional passes means **no additional observed failures** there, whatever the absolute numbers + say — and nothing more. It cannot see untested behaviour, the code paths behind the missing + dependencies, or environment-specific regressions, so the required gates still have to run in a + supported environment (CI, or a local install of the full declared dependency set) before the + change counts as verified. --- ## Where the organization's other accumulated know-how lives From a211c57fad4c3452d5afd7c3629c6d1e6ba27bd0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 22:32:39 +0000 Subject: [PATCH 25/50] docs(triage): post-#1939 noema 502 measurement; signature 12 (Pingora size contract) Signature 3: the first post-#1939 noema-review runs split 1/1; the failing run's policy report shows the diversified pool #1939 promised and still ends in a 502 after a ~548 s-per-route walk (host 1's arithmetic from source), so a base-merge push recovers pre-#1939 heads but does not shorten the post-#1939 walk; #1943/#1944/#1945 make the per-route timeline readable from the noema-sidecar-evidence artifact; the remaining lever is inside contextual-orchestrator. Signature 12: required-workflow-bootstrap exit 2 in ~5 s on "exceeds the size contract" -- the Contents API's 1 MiB inline ceiling on a patchless text file, fixed by #1946's Git Blobs API route. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- AGENTS.md | 3 +- docs/doctoring/ci-failure-signature-triage.md | 152 ++++++++++++++++++ 2 files changed, 154 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 407c74c696..3ed24d9fcf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -237,7 +237,8 @@ they only pay off when an agent other than their author reads them. unscoped `pytest` collection, stale `head_sha` notifications, queued-versus-failed, same-anchor merge conflicts, futile CodeQL compat re-runs, cancelled-versus-failed and auto-update check starvation, and - drafts that no gate will ever approve, review, or rebase), each + drafts that no gate will ever approve, review, or rebase, and the Pingora + policy's 1 MiB Contents-API ceiling on oversized text), each with its verified mechanism, the action to take, and the actions not to take, plus the rule for deciding when a re-run can help at all. Also carries the diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 1e6b9f838d..d616057648 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -104,6 +104,17 @@ updating the variable; neither is a per-PR action. **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. +**What the one handler that passed the gate did, 2026-09-06T01:34Z.** `.github` #1492's handler run +`33991725331` was dispatched by the *scheduler* path (`github-actions[bot]`, 21:00:01Z, so it passed +`validate-pr-metadata`), then queued stage by stage — `validate-pr-metadata` 22:18Z, +`coverage-source-tree` 23:15Z, `coverage-evidence` 00:30Z, `opencode-review` 01:26Z (signature 7: each +`needs:` stage re-enters the queue) — and its review reported `Model pool: exhausted` and fell back to +deterministic evidence, which returned `REQUEST_CHANGES` because a stale cancelled Strix check sat on +the head. So fixing the dispatcher variable (#1929) reopens the *path*; the verdict at the end of it +still runs into signature 3's pool, and the model-unavailable fallback can only approve a head whose +peer checks are already complete and clean. Expect the first post-fix verdicts to be +`REQUEST_CHANGES` on heads carrying any stale red check, not approvals. + --- ## 2. `strix` fails within seconds with `AttributeError: ... no attribute 'asyncio'` or `ModuleNotFoundError: No module named 'httpx2'` @@ -203,6 +214,112 @@ a new event binds the current sidecar — and it is worth doing even mid-batch, would have been reset had already failed. Applied to four heads at 21:15–21:21Z, each after the full local gate. +**Measured again 2026-09-05T22:20Z, after #1939: the fix closed the pool-diversity gap, not the 502 +itself.** The first post-#1939 `noema-review` runs (created after 17:25Z, so bound to the new sidecar) +split 1 success (`#1902`, run `33982955696`, 10 minutes) to 1 failure (`#1872`, run `33985079091`, +40 minutes of held runner). The failing run's policy report now lists `openrouter`, `nvidia_nim` and +`nvidia_nim_sub` routes as `ready` — the diversity #1939 promised — and still ended in +`HTTP Error 502 … duration=1989.9s, served_model=deepseek-ai/deepseek-v4-flash-0731`; `#1940`'s run +`33981136873` did the same over 3122 s. Host 1 traced the path from source twice and the first artifacts +settled it (`.github` #1938 thread, 01:28Z): the `attempt=1/1` lines in a trace are the **preflight +probes** (preflight client `max_retries=0`, twelve routes in about a minute); the serving phase for +the `orchestrator/free` virtual pool is `_invoke` — `tool_retry_attempts=1` on top of the client's +`max_retries=2`, so each agent gets two rounds of three attempts at the 90 s per-recv limit, measured +270.3–271.1 s per silent round and 541 s for one agent's two rounds — and an internal repair/judge +loop can re-enter `_invoke` and walk the same agents again inside one caller request. Two artifacts +show the shape. `.github` #1661's run `33995553859` (3873 s → 502): the same three agents walked +three times, 13 silent rounds of ≈270 s on the two NVIDIA deepseek-flash keys, `cohere/north-mini-code` +ending `provider_rejected_permanent`, and two preflight-ready routes (`nvidia_nim_sub` deepseek-pro, +`dots-3`) **never attempted**. `.github` #1946's run `33996197307` (artifact `9980320306`, 2539 s → +502): `nvidia_nim_sub` deepseek-v4-flash attempted 29 times and never answered (24 timeouts), +`provider_exhausted` nine times for that one route, its circuit opened twice (`failures=3.0 +threshold=3 reset_seconds=30.0`) and was re-admitted within a minute each time because a 30 s reset +is shorter than one 90 s attempt, and the last call at 01:25:10 went to the same route and timed out +while two other ready routes had answered — about 36 of 43 minutes on a route that produced no byte. +A long duration therefore counts silent ≈270 s rounds, most of them re-admissions of the same one or +two agents; it does not count distinct routes, and the per-agent timestamps in the artifact are the +only direct measure. #1943 (sidecar `DEBUG` trace), #1944 (`noema-review` +uploads `strix_runs/contextual-orchestrator-sidecar.stderr.log` and the preflight report as the +`noema-sidecar-evidence` artifact on failure) and #1945 (the sanitizer admits the orchestrator's +`provider_attempt` / `provider_exhausted` / `circuit_*` lines), all on `main` by 22:15Z, turn that +into a per-route timeline you can read from the artifact instead of inferring. A second post-#1939 +shape, 23:47Z: `.github` #1938's run `33992736660` ended after **551.0 s** with `HTTP Error 429: Too +Many Requests` and `served_model=deepseek-ai/deepseek-v4-pro-0813`. Do not read that as "one route, +no failover" — host 1 refuted exactly that reading from the pinned source within minutes: a 429 is +`retryable=True` (`provider_errors.py:82`), which `_invoke` turns into one same-agent retry and then +`FAILOVER_AGENT` to the next candidate (`orchestrator.py:7828-7868`, `tool_fallback.py:114-154`), and +the caller receives the *last* route's *last* error, so `served_model` names the last route tried, +never the first. What that run's preflight actually held: `ready 3 / rejected 9` — the three ready +routes were all NVIDIA `deepseek-v4` (`flash` on the sub account, `pro` on both accounts), and all four +`openrouter` routes were rejected at preflight with 429 (plus three NVIDIA 404s and one 529). So +#1939's account interleave delivered the diversity and OpenRouter's rate limit took it away again +before the first request: the pool the failover walked was NVIDIA-only for a different reason than +before. There is no "routes walked" fingerprint: `duration / 270` approximates silent *rounds*, and +the artifacts above show the same agent taking most of them, so duration counts re-admissions of a +stalled route, not routes. Only the `noema-sidecar-evidence` artifact +(#1944; that run predates it, and a re-run keeps the same `workflow_sha`) gives the walked-candidate +list with the time each one held the request. Tally of post-#1939 `noema-review` runs in `.github` with a PR attached, by the +`Prepare Noema model verdict` step's own conclusion, at 02:10Z: **0 succeeded, 23 failed** (the +23rd being `#1938`'s sanctioned re-run, attempt 2, 3014 s → 502 — the measurement that keeps every +other held re-run held) +(verdict-step durations 199–4273 s, median ≈1500 s; among them `#1872` 502 after 1989.9 s, `#1938` +429 after 551 s, `#1930` 429 after 1444.7 s, `#1913` 502 after 2123.7 s, `#1916` 502 after 3783.4 s — +63 minutes of one or more candidates holding the request open). Pre-#1939 (10:00–16:25Z) the same +count was 7 of 14. Two earlier versions of this tally were wrong in two different ways: one counted +"4 succeeded" +because three run-level successes at 21:59–22:15Z were the closure-event runs of #1943/#1944/#1945 +after merge, whose `noema-review` job was skipped before any step ran. A `pull_request_target` run +whose job skipped on "events without pull request context" reports `conclusion: success` at run +level; and the other counted `#1902`'s two green jobs as verdicts when they were draft skips — the +job log says `PR is draft; Noema verdict preparation skipped.` and the verdict step took 0–1 s. Count +jobs whose `Prepare Noema model verdict` step has a conclusion *and* a duration in minutes; never run +conclusions, never a green job whose verdict step finished in a second. + +**Why the ready count fell from 5–6 to 1–3 across #1939, independent of load.** The 4+4+4 fill takes +each NVIDIA key's first four models in catalog order, which is alphabetical within the tier: +`deepseek-v4-flash`, `deepseek-v4-pro`, `gemma-3-12b`, `gemma-3-4b` — and the two `gemma-3` routes are +permanent 404s, so each key serves two working routes, both the most-contended deepseek models. The +pre-#1939 8+4 fill reached `meta/llama-3.2-11b`, `meta/llama-3.2-90b` and `meta/muse-glimmer-30b`, which +were *ready* in every pre-#1939 Strix artifact opened (runs `33979406293`, `33979153466`, +`33978435797`: ready 6, 6, 5 of 12 at 16:37–16:56Z); discovery still lists 21 models per NVIDIA key +that never reach the served set. Same family as #1415 / #1921 / #1939 — an alphabetical tiebreak plus a +budget starves whatever sorts last, and this time the budget is four per account with two dead routes +sorting first. The lever proposed on `.github` #1948 (host 1's lane) is a lazy per-account fill that +probes down the ranked list until K routes are ready, which also bounds probe spend. +The strongest evidence of what the 429 variant *is* came from `.github` #1930's `strix` failure in the +same window (run `33992904674`, 23:48–00:12Z), because the Strix workflow already ships the sidecar +files in `strix-reports`: preflight **ready 1 / rejected 11** of 12 selected routes — all four +`openrouter` 429, the primary NVIDIA `deepseek-v4-pro` and `v4-flash` 429, the sub-account `v4-flash` +a `TimeoutError`, the four `gemma-3` routes 404 — and the sole ready route (`nvidia_nim_sub` +`deepseek-v4-pro-0813`) answered `429 rate_limit_exceeded` on first contact and on all five of Strix's +replays (`run.json` `llm_usage.requests: 0`). `.github` #1938's Strix run `33992736699` forty minutes +later read `ready 3 / rejected 9`, got seven completions through, then hit the same persistent 429 and +stopped after 41.5 minutes; its sidecar stderr counted 36 × `status=429 rate_limit_exceeded` and +14 × `status=500 internal_error`. `.github` #1916's Strix run `33992902189` (artifact `9980773663`) +read `ready 4 / rejected 8` — all four deepseek-v4 routes — made 29 requests over **2 h 18 min** +while the gateway answered `status=500 code=internal_error` 76 times, and died on Strix's stream +idle timeout: a third shape (bulk gateway 500s) next to the 429 and the held-open 502, and the +single most expensive review job of the night. That is not a routing defect: the free pool had no capacity on any +account at that hour, and #1939's 4+4+4 interleave had nothing to interleave. Each +such job still holds a runner for ≈25 minutes before failing, in a queue hundreds deep. **Do not +re-run a 429-variant failure while the most recent artifact in the repository shows ≤1 ready route**: +the odds are near zero and the cost is the slot. The lever (a pool that rate-limits on first contact +should fail in seconds, not minutes, and an upstream that holds a request open needs a response-start +or total deadline at the gateway — the owner's call) lives in the orchestrator's passthrough policy +and in strix-agent's replay loop — the orchestrator lane's, and an owner decision on paid routes — not in this repo's +sidecar; the owner's tracking issue is `contextual-orchestrator#1045` (fix PR `contextual-orchestrator#1049`, +failover with typed attempt evidence), and the night's cost is posted there: 8 failed `noema-review` +/ `strix` jobs on runs created 21:00–00:18Z burned **184 runner-minutes** in a 230-deep queue, and +the three that completed in the next 25 minutes (`#1938` Strix 41.5, `#1916` Noema 63, `#1930` Strix +32) took the total past **290**. So: a +base-merge push recovers a head from the +*pre-#1939* single-upstream stall, and it is still worth doing; it does not make the post-#1939 walk +shorter; and re-running a post-#1939 failure is the coin flip described above, at a cost the previous +run's duration does not predict. The remaining lever — the per-recv timeout and retry product +inside the gateway — lives in `contextual-orchestrator`, not in this repo's sidecar, and the +orchestrator lane holds it; do not add a caller-side deadline here (`noema_review_gate.py:1520` +states why the caller carries none). + **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a @@ -605,6 +722,41 @@ missing one: the PR belongs to someone else. --- +## 12. `required-workflow-bootstrap` fails in ~5 seconds with `Pingora edge policy could not establish complete evidence: … exceeds the size contract` + +**Symptom.** The organization-required `opencode-review.yml` run fails in its very first job, +before any review is dispatched, with one line: +`##[error]Pingora edge policy could not establish complete evidence: GitHub content evidence for + exceeds the size contract`, exit code 2. It recurs on every push of the same branch. Seen on +`#1678` (`automation/sbom-inventory`, run `33989047645`) where `` was `docs/sbom/inventory.json`. + +**Mechanism.** Deterministic and content-blind — it is not an Nginx finding. `scripts/ci/pingora_edge_policy.py` +reads each changed file's final content through the Contents API, which stops inlining content at +1 MiB and answers `encoding: "none"` with only a `size`. GitHub also omits the diff `patch` for a file +that large, and `_needs_content_scan` treats a missing patch as "must scan", so any text file over +1 MiB that is neither a documentation suffix (`.md`, `.txt`, …) nor a verified binary document +(`.pdf`, `.png`) goes straight to a `ContentSizeExceededError` that only the documentation-PDF path +knows how to absorb; everywhere else it fails the whole check closed. `#1678`'s inventory was +1,148,611 bytes (236 bytes on `main`) and, scanned offline with `main`'s own `scan_content`, carried +zero denied forms — it does not even contain the string `nginx`. + +**Do.** Fix the evidence route, not the file: `.github` #1946 follows the Contents response's blob +`sha` to the Git Blobs API (bounded at 11 MiB so the wrapped base64 response fits the existing 16 MiB +reader), binds the blob back to the Contents metadata, and scans the bytes like any inline file. +Until it is on `main`, an affected PR cannot pass this context by re-run, rebase, or any change that +keeps the file over 1 MiB; a base-merge push after the fix lands is what re-binds the required +workflow's trusted source. Before assuming this signature, confirm the file really is clean: the +offline reproduction is `git show :` piped into `scan_content` from `main`'s module (the +module needs `sys.modules[spec.name] = module` before `exec_module`, or its dataclasses fail to +import). + +**Do not.** Do not add the path to the documentation exemptions, do not raise `MAX_FILE_BYTES`, and +do not shrink the file to dodge the ceiling — each of those weakens or hides evidence the policy is +supposed to establish. Do not re-run the job: the same source produces the same exit 2 in the same +5 seconds, and the run only re-enters the starved queue (signature 7). + +--- + ## Absence of data flow is not evidence that an edge is safe to cut Under queue saturation every serial `needs:` hop costs a full queue round (measured at 1.89–2.76 h From 362193ddb96fef68323024479344ab6d4a70071c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 02:36:55 +0000 Subject: [PATCH 26/50] =?UTF-8?q?docs(triage):=20CodeRabbit=20round=20on?= =?UTF-8?q?=20403d114f0=20=E2=80=94=20#1902=20was=20a=20draft=20skip,=20es?= =?UTF-8?q?cape=20a=20line-leading=20#1939?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 22:20Z paragraph of signature 3 still counted #1902's green job as a noema-review success; it was a draft skip with a 1 s verdict step, as the stage-level tally further down already says. Wrap the line-leading #1939 reference in inline code (MD018). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index d616057648..65b18b2bdd 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -216,8 +216,10 @@ local gate. **Measured again 2026-09-05T22:20Z, after #1939: the fix closed the pool-diversity gap, not the 502 itself.** The first post-#1939 `noema-review` runs (created after 17:25Z, so bound to the new sidecar) -split 1 success (`#1902`, run `33982955696`, 10 minutes) to 1 failure (`#1872`, run `33985079091`, -40 minutes of held runner). The failing run's policy report now lists `openrouter`, `nvidia_nim` and +were 1 failure (`#1872`, run `33985079091`, 40 minutes of held runner) and one green job that was +not a verdict at all: `#1902`, run `33982955696`, was a draft skip (`PR is draft; Noema verdict +preparation skipped.`, verdict step 1 s). The first version of this paragraph counted it as a +success; the stage-level tally further down is the authoritative one. The failing run's policy report now lists `openrouter`, `nvidia_nim` and `nvidia_nim_sub` routes as `ready` — the diversity #1939 promised — and still ended in `HTTP Error 502 … duration=1989.9s, served_model=deepseek-ai/deepseek-v4-flash-0731`; `#1940`'s run `33981136873` did the same over 3122 s. Host 1 traced the path from source twice and the first artifacts @@ -252,7 +254,7 @@ the caller receives the *last* route's *last* error, so `served_model` names the never the first. What that run's preflight actually held: `ready 3 / rejected 9` — the three ready routes were all NVIDIA `deepseek-v4` (`flash` on the sub account, `pro` on both accounts), and all four `openrouter` routes were rejected at preflight with 429 (plus three NVIDIA 404s and one 529). So -#1939's account interleave delivered the diversity and OpenRouter's rate limit took it away again +`#1939`'s account interleave delivered the diversity and OpenRouter's rate limit took it away again before the first request: the pool the failover walked was NVIDIA-only for a different reason than before. There is no "routes walked" fingerprint: `duration / 270` approximates silent *rounds*, and the artifacts above show the same agent taking most of them, so duration counts re-admissions of a From bd414bef7f1ea4bffb2b99afda9c10dc9eb3e278 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 03:10:18 +0000 Subject: [PATCH 27/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20the=20retry-stacking=20loop=20is=20fixed=20by=20the=20sideca?= =?UTF-8?q?r=20pin=20bump=20(efb892692=20/=20co#1081)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Held for the next batched push of this PR. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 65b18b2bdd..dc88330910 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -322,6 +322,17 @@ inside the gateway — lives in `contextual-orchestrator`, not in this repo's si orchestrator lane holds it; do not add a caller-side deadline here (`noema_review_gate.py:1520` states why the caller carries none). +**Closed again 2026-09-06T03:01Z: the retry-stacking loop itself is fixed on `main`.** The owner +bypass-merged `.github` `efb892692`, advancing the sidecar's `ORCHESTRATOR_PIN_SHA` from `2e414d15` +to `414f2297` — the commit that merges `contextual-orchestrator#1081`, whose fix stops `_invoke`'s +own retry-then-failover decision from stacking on top of the client's `max_retries` (reproduced on +the old pin as six real attempts per candidate, confirmed at ≤ 2 on the new one). The bypass was +necessary for the same reason as every sidecar pin bump: the PR's own required reviews run the base +branch's still-stale sidecar (`pull_request_target` trust boundary), so normal review would have hit +the bug being fixed. As with #1939, a head whose runs failed before the bump does not recover by +re-run — one base-merge push binds the new pin — and the pool condition (signature 3's rate-limited +free tier, `.github` #1948) is a separate lever that this bump does not touch. + **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a From 7152d9f421ba6835a4c60d6f6bf0510cca09f0f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 04:38:55 +0000 Subject: [PATCH 28/50] =?UTF-8?q?docs(triage):=2004:30Z=20measurements=20?= =?UTF-8?q?=E2=80=94=20the=20fail-fast=200/12=20provisioning=20shape=20(si?= =?UTF-8?q?gnature=203),=20the=20slot=20hold=20with=20the=20pool=20at=20ze?= =?UTF-8?q?ro=20and=20the=20CodeQL=20dispatch=20flood=20(signatures=207=20?= =?UTF-8?q?and=201)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index dc88330910..1fd3e7d941 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -99,7 +99,10 @@ verdict can ever be published for any head (#1929). Discriminator: open the newe your head — if its **first** job concluded `failure` with `::error::repository_dispatch authorization rejected actor=…`, you are in this case, and no amount of waiting or re-running the required job changes it. Remedy is #1932 (a comma-separated allowlist, mechanism only) plus an owner -updating the variable; neither is a per-PR action. +updating the variable; neither is a per-PR action. The same variable gates +`codeql-scan-dispatch.yml:142`, and the sibling repositories re-dispatch after each rejection, so the +flood is visible from any repository's queue: 01:30–04:30Z on 2026-09-06, 231 `CodeQL Scan Dispatch` +runs, two or three per SHA, every one rejected in 3–6 s after ≈65 minutes of queue (signature 7). **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. @@ -333,6 +336,19 @@ the bug being fixed. As with #1939, a head whose runs failed before the bump doe re-run — one base-merge push binds the new pin — and the pool condition (signature 3's rate-limited free tier, `.github` #1948) is a separate lever that this bump does not touch. +**A fourth shape, fail-fast at provisioning (first seen 2026-09-06T04:24Z).** When preflight finds +*zero* ready routes the launcher exits before `/healthz`, `contextual_orchestrator_review_sidecar.sh` +reports `sidecar exited before healthz (status 1)`, and the job fails in ≈5 minutes at "Provision +contextual-orchestrator review sidecar"; "Prepare Noema model verdict" is *skipped*, and the +`noema-sidecar-evidence` artifact still uploads. `.github` #1913's run `34006939646` (artifact +`9982569956`; pre-bump pin, the run having been created 24 minutes before `efb892692`) read +`ready 0 / rejected 12`: both keys' `deepseek-v4-flash` 429, both keys' `deepseek-v4-pro` — until +then the route carrying reviews — `TimeoutError` at the 90 s probe bound, the four `gemma-3` 404, +all four OpenRouter free routes 429, bytez discovery `http_status_500`. Tally it apart from +verdict-step failures (a skipped verdict, not a failed one) and read it as the cheapest form of this +signature: five minutes of runner instead of thirty to seventy. The re-run rule above applies +unchanged — 0 ready is the strongest possible ≤1 reading — and the base-merge remedy is the same. + **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a @@ -519,6 +535,23 @@ tool's `cancel_workflow_run` method and verify each run with `actions_get` `get_ Never apply this to PR scans — a PR scan is consumed by its check, and the PR-scoped group already retires superseded heads on its own. +**Second application, 2026-09-06T04:36Z, with the pool at zero.** `.github` had 170 queued runs +against 10 in progress. Seven of the ten were `strix` jobs on the pre-bump sidecar holding runners +since 00:10–03:23Z (the five oldest 3 h 45 m–4 h 25 m inside "Run Strix (quick)", signature 3's +bulk-500 shape, with no job timeout by design — `strix.yml:345-350`, `noema-review.yml:261-276` — so +such a job holds its slot until the 6-hour Actions maximum), two were `noema-review` verdict steps +started 03:22Z / 03:47Z, and nothing created after the 03:01Z pin bump had started (11 `noema-review` +runs queued since 03:07Z). Of the 170 queued, 85 were `CodeQL Scan Dispatch` handlers from sibling +repositories rejected at the same actor gate as signature 1's (`codeql-scan-dispatch.yml:142`, +`actor=opencode-agent[bot]` against `github-actions[bot]`): each waits ≈65 minutes for a slot, fails +in 3–6 s, and the source repository re-dispatches — 231 runs in three hours, two or three per SHA — +so their cost is queue position, not runner-minutes. The two `push`/`main` scans (`d9eb9f79b`, +`972b74be2`; both ancestors of `main@efb892692`, whose own push scan was queued) passed the three-part +test and were retired (`33994595180`, `33995072470`, both `completed/cancelled` within 20 s); the +five PR scans were left alone. Note what freeing a slot buys while the pool reads 0 of 12: the next +queued review fails in five minutes with an artifact (signature 3's fail-fast shape) instead of +holding a runner for hours without one — still the right outcome. + **Do.** Read each check's actual conclusion. Truly `queued` → leave it and work elsewhere. A workflow that was never assigned a runner → escalate on `.github` #712, #1531, #1219. If a PR of yours has never completed a review cycle, count your own pushes to it before blaming the queue: batch your From ef151f02f4c7e3c6b20c1d27d4e14d02089bce96 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 04:54:54 +0000 Subject: [PATCH 29/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20the=20fail-fast=20shape=20recurs=20on=20the=20new=20pin=20(#?= =?UTF-8?q?1661),=20and=20the=20two=204=20h=2015=20m=20pre-bump=20Strix=20?= =?UTF-8?q?jobs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 1fd3e7d941..85a2122981 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -316,7 +316,11 @@ sidecar; the owner's tracking issue is `contextual-orchestrator#1045` (fix PR `c failover with typed attempt evidence), and the night's cost is posted there: 8 failed `noema-review` / `strix` jobs on runs created 21:00–00:18Z burned **184 runner-minutes** in a 230-deep queue, and the three that completed in the next 25 minutes (`#1938` Strix 41.5, `#1916` Noema 63, `#1930` Strix -32) took the total past **290**. So: a +32) took the total past **290**. Two more pre-bump Strix jobs then ran to the shape's natural end: `#1271` +(`33995516908`) and `#1231` (`33994984527`) each held a runner for 4 h 15 m (preflight `ready 6 / 12` +at 00:2xZ; 206 and 202 gateway `status=500 code=internal_error` lines; #1271 issued 170 requests, 136 +of them with usage, and ended `failed`) — about 510 runner-minutes for no verdict, the two most +expensive review jobs on record. So: a base-merge push recovers a head from the *pre-#1939* single-upstream stall, and it is still worth doing; it does not make the post-#1939 walk shorter; and re-running a post-#1939 failure is the coin flip described above, at a cost the previous @@ -347,7 +351,28 @@ then the route carrying reviews — `TimeoutError` at the 90 s probe bound, the all four OpenRouter free routes 429, bytez discovery `http_status_500`. Tally it apart from verdict-step failures (a skipped verdict, not a failed one) and read it as the cheapest form of this signature: five minutes of runner instead of thirty to seventy. The re-run rule above applies -unchanged — 0 ready is the strongest possible ≤1 reading — and the base-merge remedy is the same. +unchanged — 0 ready is the strongest possible ≤1 reading — and the base-merge remedy is the same. It recurred on the new pin thirty minutes later — `.github` #1661's run `34008191123` +(artifact `9982909775`, sidecar at `414f2297`, 04:48–04:51Z) read the identical `ready 0 / rejected +12` — so the pin bump cannot be evaluated until the pool has at least one ready route: the first +post-bump tally line is a provisioning failure, not a verdict. Strix shows the same shape at its own step, "Provision +contextual-orchestrator Strix sidecar", in about nine minutes (`.github` #1938's run `34008403183`, +artifact `9983066996`, 05:05Z — the third consecutive 0 / 12 artifact); "Run Strix (quick)" is +skipped, and the `strix-reports` artifact carries the preflight and discovery evidence. + +**A fifth shape — one ready route that cannot serve (first seen 2026-09-06T05:20Z, the first +post-#1081 serving-phase sample).** Route preflight passes with `ready 1`, "healthz and +provider-route preflight confirmed", and then the sidecar script's *gateway* preflight +(`contextual_orchestrator_review_sidecar.sh`, `REVIEW_PREFLIGHT_GATEWAY_MAX_ATTEMPTS=3`) cannot get +one completion through that route: each attempt is served with exactly two 90 s tries on the only +candidate (`1 + tool_retry_attempts`, no transport-retry stacking underneath — the post-#1081 +arithmetic), ends `request_failed status=502 code=provider_connection_error`, and the job fails at +the provisioning step after 3 × 2 × 90 s ≈ 9 minutes of serving on top of the route walk. `.github` +#1946's run `34008655765` (artifact `9983259344`): `nvidia_nim_sub` `deepseek-v4-pro` answered the +16-token probe in 88 s, then timed out six times (circuit `failures=1.0 → 3.0`, `circuit_opened`, +re-admitted after the 30 s reset); 14 min 4 s in total. Read it as capacity, not routing: a probe +answered at 88 s is inside the deadline and outside any usable budget. On the old pin the same +failure cost 3 × 6 × 90 s. The re-run rule's boundary case: the artifact shows exactly one ready +route, and it is the route that just failed — do not re-run on that evidence. **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you From 199336dbe4419ba174f91e49c43e54a02916f4cc Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 06:54:24 +0000 Subject: [PATCH 30/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20the=20first=20post-#1081=20Strix=20verdict:=20passthrough=20?= =?UTF-8?q?timeouts=20never=20trip=20the=20breaker=20(co#1082)=20and=20the?= =?UTF-8?q?=2030=20s=20reset=20against=20a=2090=20s=20attempt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 85a2122981..654babf34a 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -374,6 +374,23 @@ answered at 88 s is inside the deadline and outside any usable budget. On the ol failure cost 3 × 6 × 90 s. The re-run rule's boundary case: the artifact shows exactly one ready route, and it is the route that just failed — do not re-run on that evidence. +**What the first post-#1081 Strix verdict looked like (2026-09-06T06:47Z), and the residual it +isolates.** `.github` #1930's run `34008575120` (artifact `9984330203`, pin `414f2297`) provisioned +with `ready 4 / rejected 8` at 05:17Z, then spent 86 minutes in "Run Strix (quick)": Strix completed +8 requests with usage while the gateway logged 48 `TimeoutError`s and 48 × `status=500 +code=internal_error`, 44 of them on the first-ranked `nvidia_nim` `deepseek-v4-flash` at 90 s each, +until Strix's stream idle timeout (`strix/config/models.py:371`) ended the scan as +`STRIX_PROVIDER_UNAVAILABLE`. The discriminating count: **0 of the 48 timeouts is followed by a +`circuit_failure` record, while 6 of the 14 fast HTTP failures are** (and those failed over). The +stacking #1081 removed is gone (`attempt=1/1` throughout, no `2/3`); what remains is the tool-bearing +passthrough class of `contextual-orchestrator#1082` — a bare `TimeoutError` re-raised as `500 +internal_error` with `_record_failure` unreachable, so the stalled route is first again on every +retry — plus a breaker whose 30 s reset restores a clean count against a 90 s attempt +(`orchestrator.py:8031-8046` at the pin), so even a recorded stall costs about 4.5 of every 5 +minutes. Count a run of this shape as *serving-stall (passthrough timeout)*: preflight passed, +requests completed, the slot was spent on one silent route. A re-run is a coin flip on whether that +route answers; hold it while the route is the same one the artifact names. + **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you have before acting. This failure is *runtime-external*: the pinned source is fine and simply made a From b7fed12a93ac0c46b3975e45fc5b69bc32bf349e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 07:25:57 +0000 Subject: [PATCH 31/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20second=20post-#1081=20Strix=20sample=20(#1916:=2063=20timeou?= =?UTF-8?q?ts,=200=20circuit=20records,=20126=20min)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 654babf34a..30f265abc8 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -389,7 +389,11 @@ retry — plus a breaker whose 30 s reset restores a clean count against a 90 s (`orchestrator.py:8031-8046` at the pin), so even a recorded stall costs about 4.5 of every 5 minutes. Count a run of this shape as *serving-stall (passthrough timeout)*: preflight passed, requests completed, the slot was spent on one silent route. A re-run is a coin flip on whether that -route answers; hold it while the route is the same one the artifact names. +route answers; hold it while the route is the same one the artifact names. The second sample, an hour +later, scaled the same way: `.github` #1916's run `34008489633` (artifact `9984863327`) completed 42 +requests (39 with usage, ~2 M input tokens) over 126 minutes while logging 63 timeouts, 63 × 500, +114 attempts on the same first-ranked route and 0 circuit records after a timeout — the scan that +did the most work on this pool and still could not finish. **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you From aab3a2ff3467a2961d39820df1eef9b4916a12b8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 08:05:46 +0000 Subject: [PATCH 32/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20third=20post-#1081=20Strix=20sample=20(#1946)=20closes=20the?= =?UTF-8?q?=20set?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 30f265abc8..25c682356e 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -393,7 +393,10 @@ route answers; hold it while the route is the same one the artifact names. The s later, scaled the same way: `.github` #1916's run `34008489633` (artifact `9984863327`) completed 42 requests (39 with usage, ~2 M input tokens) over 126 minutes while logging 63 timeouts, 63 × 500, 114 attempts on the same first-ranked route and 0 circuit records after a timeout — the scan that -did the most work on this pool and still could not finish. +did the most work on this pool and still could not finish. The third (`.github` #1946, run +`34008655751`, artifact `9984885189`: 32 requests, 65 timeouts, 99 attempts on that route, 0 circuit +records, 126 minutes) closed the set: three scans, one class, 176 timeouts, about 14 runner-hours of +Strix on this pool since 00:10Z for no verdict. **Why a re-run is the right remedy here and the wrong one for signature 2.** These two failures look alike — a red required check on a review job — and take opposite actions, so check which one you From b36876bf9382d1df5db56a3e016035a509f89725 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 08:20:30 +0000 Subject: [PATCH 33/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20split=20Strix=20provider-unavailable=20verdicts=20on=20the?= =?UTF-8?q?=20sandbox=20class=20(#1953,=20host=201's=20finding)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 25c682356e..43e26d9bf9 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -388,7 +388,13 @@ internal_error` with `_record_failure` unreachable, so the stalled route is firs retry — plus a breaker whose 30 s reset restores a clean count against a 90 s attempt (`orchestrator.py:8031-8046` at the pin), so even a recorded stall costs about 4.5 of every 5 minutes. Count a run of this shape as *serving-stall (passthrough timeout)*: preflight passed, -requests completed, the slot was spent on one silent route. A re-run is a coin flip on whether that +requests completed, the slot was spent on one silent route. Before counting any Strix +`STRIX_PROVIDER_UNAVAILABLE` verdict here, split on `gate-console.log`: a run whose console shows +`Docker image ready` and then `loginAsGuest failed after 10 attempts` with an exit after ≈240 s is a +Strix-internal sandbox outage (the Caido proxy on `127.0.0.1:48080` never accepted), not the gateway +— host 1 found two of six recent `strix-reports` artifacts in this class on 2026-09-06, and #1953 +gives it its own bounded retry and a verdict that names it (`STRIX_SANDBOX_UNAVAILABLE`). The three +scans above carry no such line. A re-run is a coin flip on whether that route answers; hold it while the route is the same one the artifact names. The second sample, an hour later, scaled the same way: `.github` #1916's run `34008489633` (artifact `9984863327`) completed 42 requests (39 with usage, ~2 M input tokens) over 126 minutes while logging 63 timeouts, 63 × 500, From 36a6a52af1a64a990a8b4f39512dcec1e588a9d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 10:20:40 +0000 Subject: [PATCH 34/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20the=20pool=20is=20hourly-intermittent,=20phase=3Dvalidating?= =?UTF-8?q?=20is=20its=20own=20class,=20and=20the=20measured=20cost=20of?= =?UTF-8?q?=20the=20#1947/#1949=20walk?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 43e26d9bf9..48bcd0feec 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -394,7 +394,27 @@ requests completed, the slot was spent on one silent route. Before counting any Strix-internal sandbox outage (the Caido proxy on `127.0.0.1:48080` never accepted), not the gateway — host 1 found two of six recent `strix-reports` artifacts in this class on 2026-09-06, and #1953 gives it its own bounded retry and a verdict that names it (`STRIX_SANDBOX_UNAVAILABLE`). The three -scans above carry no such line. A re-run is a coin flip on whether that +scans above carry no such line. + +**Two measurements from 2026-09-06 that change how to read this signature.** First, the pool is +intermittent by the hour, not absent: at 08:26Z `contextual-orchestrator#1044`'s `noema-review` +(run `34018735728`, artifact `9986228615`) preflighted `ready 6 / probed 16 / skipped 4 / deferred 2` +— both NVIDIA keys' flash and pro *and* both keys' `llama-3.2-11b` — and carried a 21-minute review +to a served completion; at 09:50Z the same pool read `ready 0 / probed 6 / skipped 18` (`.github` +#1913, artifact `9987026530`). Second, that 08:26Z run failed at `phase=validating` +(`Noema approve cannot contain a confirmed adversarial probe`, `served_model` deepseek-v4-flash), +not `response_error`: the gateway answered and Noema's own gate rejected the model's *content*. +Tally a `phase=validating` outcome as model-output quality, in neither the capacity nor the +passthrough bucket. The same artifact gives the cheapest discriminator between the two failure +paths: **15 timeouts, 9 followed by a `circuit_failure` record** on Noema's no-tools `_invoke` path, +against 176 timeouts and 0 records on the morning's tool-bearing Strix passthrough runs. Check that +adjacency before attributing a stall. + +**What #1947/#1949 changed, measured.** On a fully rate-limited hour the walk now stops after the +per-account 429 streak instead of probing every candidate: `.github` #1913's 09:50Z run probed 6, +skipped 18, and failed provisioning in **126 s**, against 300–844 s for the same zero-ready outcome +before those PRs merged. A dead hour costs two minutes of runner rather than five to fourteen; the +evidence artifact still uploads, with `skipped_count` separate from the unreached tail. A re-run is a coin flip on whether that route answers; hold it while the route is the same one the artifact names. The second sample, an hour later, scaled the same way: `.github` #1916's run `34008489633` (artifact `9984863327`) completed 42 requests (39 with usage, ~2 M input tokens) over 126 minutes while logging 63 timeouts, 63 × 500, From d06e8ea026c2343060a2918b1f209870c0f67f1d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 13:09:41 +0000 Subject: [PATCH 35/50] =?UTF-8?q?docs(triage):=20signature=203=20=E2=80=94?= =?UTF-8?q?=20minute-scale=20pool=20alternation,=20circuit=20adjacency=20a?= =?UTF-8?q?s=20the=20path=20discriminator,=20and=20phase=3Dvalidating=20vs?= =?UTF-8?q?=20response=5Ferror?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 48bcd0feec..857353fc19 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -414,7 +414,30 @@ adjacency before attributing a stall. per-account 429 streak instead of probing every candidate: `.github` #1913's 09:50Z run probed 6, skipped 18, and failed provisioning in **126 s**, against 300–844 s for the same zero-ready outcome before those PRs merged. A dead hour costs two minutes of runner rather than five to fourteen; the -evidence artifact still uploads, with `skipped_count` separate from the unreached tail. A re-run is a coin flip on whether that +evidence artifact still uploads, with `skipped_count` separate from the unreached tail. + +**The pool alternates minute to minute, not hour to hour — do not generalise from one artifact.** +Measured on 2026-09-06 across four heads carrying the merged launcher: `.github` #1938 preflighted +`ready 6` at 11:40Z and its sibling `strix` job read `ready 0` at 11:42Z; #1946 produced three +outcomes in 45 minutes (`ready 0` at 12:00Z, a successful provision then `HTTP Error 503` at +12:05Z, and the actor-gate wait); #1916 provisioned `ready 6` at 11:55Z after failing `ready 0` +minutes earlier. A zero-ready artifact is evidence about the minute it was taken, nothing more, so +"the pool is down this hour" is not a conclusion the data supports — check the timestamp of the +artifact you are reading before attributing anything to capacity. + +**The cheapest discriminator between the two serving failure paths is circuit adjacency.** In the +same day's artifacts, Noema's no-tools `_invoke` path recorded 9 of 14 and 10 of 15 timeouts as +`circuit_failure` (and failed over), while Strix's tool-bearing passthrough recorded **0 of 21** +(`.github` #1916 run `34027314569`), 0 of 48, 0 of 63 and 0 of 65. Count `circuit_failure` lines +within two lines of each `provider_attempt_failed … TimeoutError` before attributing a stall: a +zero ratio is `contextual-orchestrator#1082`'s class (the stalled route is never recorded, so it is +re-selected on every retry), a high ratio is a route walk that ran out of candidates. + +**A verdict-step failure is not automatically a gateway failure.** Read the `phase=` field: +`response_error` is the gateway (502/503 after the caller's single attempt), while `validating` means +the model answered and Noema's own gate rejected the content — `contextual-orchestrator#1044`'s +08:26Z run failed `Noema approve cannot contain a confirmed adversarial probe` after a served +completion. Tally the two separately; only the first belongs in a capacity census. A re-run is a coin flip on whether that route answers; hold it while the route is the same one the artifact names. The second sample, an hour later, scaled the same way: `.github` #1916's run `34008489633` (artifact `9984863327`) completed 42 requests (39 with usage, ~2 M input tokens) over 126 minutes while logging 63 timeouts, 63 × 500, From e54bee4d8c1145e5c6c52af60c5c19751555b1c9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 16:34:18 +0000 Subject: [PATCH 36/50] docs(triage): the created_at rule, post-#1957 evidence, and three corrections Section 0 gains the CI-run counterpart of the stale-checkout rule: a pull_request_target run executes the base branch's scripts as of the run's created_at, not its started_at, and on 2026-09-06 those were routinely 90 minutes apart. #1916's 12:00Z preflight looked like #1957 failing 23 minutes after it merged; the run was created at 10:24Z. Classify by created_at, or by the report's own version marker (postponed_probed_count). Section 3 records what #1957 changed, measured on this repository's own heads (ready 0 -> 1, probed 6 -> 16), and three things the earlier text got wrong or did not have: - caller attempts=1 counts sidecar->gateway calls, not routing. One such attempt carried ~20 provider_attempt lines across four agents, so it is not evidence that failover did not happen. - The ~90 s attempt is now measured: 90.090 s and 90.115 s TimeoutError probes of google/gemma-4-31b-it on both NVIDIA keys. - A third reason the breaker cannot exclude a bad route, stronger than the 30 s reset: _record_success pops the circuit state entirely, so any single success zeroes the count and an alternating route never reaches the threshold. The sandbox split is refined: is_caido_bootstrap_timing_error() inspects only the last attempt, so grepping an artifact for loginAsGuest is not sufficient. Four artifacts from one day show every combination, including #1930's run 34027404208, which carries the line but ends in gateway exhaustion. Classify on the verdict line; the check annotation is generic by design. Section 7 gains the occupancy half of the queue shortage: #1930's strix job held a runner for 351.8 minutes and produced no verdict, which is execution rather than queue wait -- its sibling jobs finished in tenths of a minute. Capping long jobs is a different remedy from adding capacity. Section 9 gains the DISPATCH_OUTCOME/VERDICT_STATE discriminator that separates it from signature 1, plus the 34-of-34-queued measurement. Gate: 2956 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 118 +++++++++++++++++- 1 file changed, 117 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 857353fc19..e3df15385e 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -66,6 +66,26 @@ line number you did not read from `origin/main`. **Do not.** Do not report a finding, open a PR, or file a gap entry whose evidence came only from an unverified working tree. +**The same staleness applies to CI runs, and it is the easier trap.** A `pull_request_target` run +executes the *base branch's* trusted scripts as they were when the run was **created**, not when it +started. Under the queue depths of 2026-09-06 those two timestamps were routinely 90 minutes apart. +Worked example: `#1957` (postpone a rate-limited account's candidates instead of banning them) merged +at 11:37:30Z; `.github#1916`'s `noema-review` executed at 12:00Z — 23 minutes later — and produced the +exact pre-fix report the PR removes. The run was **created at 10:24:07Z** and waited 94 minutes for a +runner, so it ran the old launcher. Read at face value it says the fix did not work. + +**Do.** Classify a failure against the code implied by the run's `created_at`, never its +`started_at` or `completed_at`: + +```bash +curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \ + ".../actions/runs/" | jq '{created_at, run_started_at}' +``` + +A merged fix reaches only runs created after the merge. Where a report carries a version marker, use +it in preference to the timestamp: a pre-`#1957` preflight has no `postponed_probed_count` key and +stops at `probed_count 6`; a post-`#1957` one carries the key and spends the full 16-probe budget. + --- ## 1. `opencode-review` fails with "No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head" @@ -456,6 +476,73 @@ is a guaranteed no-op. The discriminator is whether the fix you are waiting on l (signature 2 — you need a new run) or in the transient behaviour of an external service (this signature — a re-run is exactly right). +**What `#1957` changed, measured on this repository's own heads (2026-09-06).** The fix works, and it +moves the failure rather than removing it. Same helper, same repository, one pre-fix and one post-fix +preflight artifact: + +| | `#1916` 12:00Z (pre-fix) | `#1913` 15:00Z (post-fix) | +|---|---|---| +| `probed_count` | 6 | **16** (full budget) | +| `postponed_probed_count` | *(key absent)* | **10** | +| `deferred_count` | 0 | **8** | +| `rejected_count` | 6 | 7 | +| `skipped_count` | 18 | 8 | +| `ready_count` | **0** | **1** | + +Transient 429s now defer instead of banning the account, the postponed tail is probed, and the walk +finds a route where it previously gave up at zero. Across `noema-review` runs *created* after the +merge, the `ready 0` provisioning shape disappeared from the terminal outcomes entirely; every +residual failure was a serving failure. So after 2026-09-06, a `ready 0` report is itself evidence the +run predates `#1957` — see section 0. + +**The residual is `target_ready 8` against `ready_count 1`, and one route is not a failover set.** +On `#1913`'s run the single ready route (`nvidia_nim / meta/llama-3.2-11b-vision-instruct`) was +attempted eleven times over 105 s, recorded one `circuit_failure`, then failed over into the +*deferred* set — candidates already known to be rate-limited — and ended +`request_failed status=429 code=rate_limit_exceeded`. + +**Read `caller attempts=1` correctly: it counts sidecar→gateway calls, not routing.** The same run +logged roughly twenty `provider_attempt` lines across four distinct agents inside that one caller +attempt. `caller attempts=1` therefore says nothing about whether failover happened; the gateway +fails over internally, and the annotation is measuring the wrong layer to answer that question. Use +the sidecar log, not the annotation, to decide whether a route was walked. + +**The ~90 s attempt figure is now measured, not estimated.** Preflight probes of +`google/gemma-4-31b-it` on both NVIDIA keys ended in `TimeoutError` at **90.090 s** +(14:56:49.978→14:58:20.068) and **90.115 s** (14:58:20.091→14:59:50.206). That is the number that +makes `circuit_reset_seconds = 30.0` too short to keep a stalled route excluded. + +**A third reason the breaker cannot exclude a bad route, stronger than the 30 s reset.** +`circuit_cleared` comes from `_record_success` (`orchestrator.py:8073-8077` at pin `414f2297`), which +does not decrement the counter — it **pops the agent's circuit state entirely** +(`cleared = self._circuit.pop(agent_id, None)`). With `circuit_failure_threshold = 3`, any single +success zeroes the accumulated count, so a route that alternates failure and success — what an +overloaded provider does — never reaches three and is never excluded at all, however long each +failure runs. Measured on `contextual-orchestrator#1043`'s run: `circuit_failure … failures=1.0` at +16:07:49.665, `circuit_cleared` at 16:07:57.361, `circuit_failure … failures=1.0` again at +16:08:46.896 — the count genuinely restarted from zero 49 seconds later. Reproduced three times on +2026-09-06 across three separate PRs (`contextual-orchestrator#1043`, `.github#1913`, `.github#1938` +at 17:59:55 → 18:00:36 → 18:01:55), each time on the single ready route the post-`#1957` preflight +had found, so it is a property of the breaker rather than a reading of one run. + +**Refinement to the sandbox split above: grepping the artifact for `loginAsGuest` is not sufficient.** +`is_caido_bootstrap_timing_error()` inspects only the **last** attempt's log, so an earlier sandbox +failure can coexist with a gateway-terminal verdict. Three of this session's own artifacts show all +three combinations: `#1938`'s run `34035203943` carries the line *and* the +`STRIX_SANDBOX_UNAVAILABLE` verdict with a healthy `ready 6` preflight and `Cost $0.0000 · Tokens 0` +(true sandbox class); `#1916`'s run `34027314569` carries the line but its terminal verdict is +`orchestrator/free exhausted` (mixed — count it as gateway); `#1946`'s run `34027492927` carries no +such line at all (gateway). **Classify on the `STRIX_PROVIDER_UNAVAILABLE:` / `STRIX_SANDBOX_UNAVAILABLE:` +verdict line, never on a grep.** A fourth specimen closed the set the same day: `#1930`'s run +`34027404208` carries `loginAsGuest` in `strix-pr-scope-jibt1j_66a8/strix.log` yet ends +`STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted`, with a healthy +`ready 6 / probed 16 / escalations_used 2` preflight — a grep files it as a sandbox outage, the +verdict line files it as gateway exhaustion, and the verdict line is right. Note also that the +generic check annotation is generic *by design*: +`strix.yml:1008` emits one `STRIX_PROVIDER_UNAVAILABLE` title for every provider-side verdict, and +`strix_quick_gate.sh:4386` deliberately keeps that token while appending the discriminator, so the +annotation alone never tells you which class you have. + **Do not.** Do not blame `served_model`; it is merely the last candidate that failed (`orchestrator.py:7812`). Do not diagnose it as a provider hang specifically — a 502 here equally covers upstream 5xx, TLS, DNS and connection failures. Do not add `timeout-minutes` to @@ -584,6 +671,17 @@ delay routinely exceeds 8 hours, and some runs queued since 2026-08-19 never sta no 4-hour upper bound. The organization-wide ceiling is saturated: 55 of 60 running jobs, with `.github` holding ~28% of the waiting volume while receiving ~4% of the execution slots. +**The other half of the shortage is occupancy, not just arrival rate — and one job can be enormous.** +Measured 2026-09-06 on `#1930`'s run `34027404208`: the `strix` job held a runner from 11:58:00Z to +17:49:47Z, **351.8 minutes**, and ended with no verdict. That is execution, not queue wait — the +run's other jobs (`Detect changed scope`, `Admit current pull request head`, +`cancel-superseded-pr-runs`) each completed in tenths of a minute at 11:20Z, so this job was assigned +and running for the whole span. Against a 10–30 minute normal scan and a shared 60-job ceiling, one +such job costs roughly a twelfth of the organization's concurrent capacity for a working day. A +second sample the same day ran 2 h 46 m (`#1519`, run `34030605428`). When triaging a queue +shortage, measure per-job `started_at`→`completed_at` on the longest-running jobs as well as the +queued count; capping or bounding those is a different remedy from adding capacity. + **Root cause: livelock, not merely slowness.** Of the 100 most recent completed `opencode-review.yml` runs, **100 were `cancelled`**; the last review to actually run to completion did so at `2026-09-04T10:14:59Z`. Median run lifetime is **10.8 minutes** against the **4.7+ hours** a run needs @@ -771,9 +869,27 @@ gh api "repos///actions/runs?per_page=20" \ If those runs are sitting `queued`, treat the compat failure as blocked on runner capacity (signature 7), not on this PR, and say so on the PR rather than retrying. +**Read the job's own two variables first; they separate this from the authorization failure in +signature 1.** The compat job prints both before it exits: + +``` +LANGUAGE: actions +DISPATCH_OUTCOME: success <- the dispatch was ACCEPTED +VERDICT_STATE: pending <- no terminal verdict exists yet +``` + +`DISPATCH_OUTCOME: success` rules out the actor-gate rejection: that failure never reaches this job's +dispatch step at all. With `success` + `pending` you are in this signature, and the question is only +whether the dispatched scan ever starts. Measured 2026-09-06 on `.github`: of the +`codeql-scan-dispatch.yml` runs created in one hour from 13:50Z, **34 of 34 were `queued` and 0 had +completed**, including the run this PR's own compat job had just created seconds earlier. One comment +on the PR naming that count is the whole remedy; there is nothing to fix in the diff. + **Do not.** Do not re-run while the dispatched scans are queued: it is pure queue amplification, and it makes the shared condition worse for every other PR in the organization. Do not read the fast -failure as a defect in the PR's code — the job never got as far as analysing it. +failure as a defect in the PR's code — the job never got as far as analysing it. Do not report it as +signature 1 without checking `DISPATCH_OUTCOME` — the two produce the same red check and take +different actions. --- From 4bcf71c6695c3ead05fab115ff0b3b2a8c414e06 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 01:20:39 +0000 Subject: [PATCH 37/50] docs(triage): correct the Strix verdict-line test and add five measured signatures The catalogue told a reader to classify a Strix failure on "the STRIX_PROVIDER_UNAVAILABLE: / STRIX_SANDBOX_UNAVAILABLE: verdict line". Written with a slash, that reads as two alternative tokens, and a reader who implements it as a prefix test misfiles every sandbox run: on post-#1953 main the sandbox verdict is one composite line beginning "STRIX_PROVIDER_UNAVAILABLE: STRIX_SANDBOX_UNAVAILABLE: ..." (strix_quick_gate.sh:4393), the leading provider token kept on purpose so the workflow still classifies a finding-free sandbox outage as incomplete infrastructure evidence. The test is a substring search, which is what the one correct consumer does (opencode-review-dispatch.yml:5648). Runs created before #1953 merged (2026-09-06T07:43:39Z) cannot carry the token at all, so their verdict line is silent about the class. contextual-orchestrator#983's strix job 100891196281 is the specimen: all three attempts died in "loginAsGuest failed after 10 attempts: curl exit 7 ... 127.0.0.1 port 48080", the loop announced "Retrying model 'orchestrator/free' due to Caido sandbox bootstrap timing (attempt 3/3)", and the terminal verdict was still the bare gateway sentence -- the exact corruption #1953 was written to end. For such a run the faithful signal is the retry-reason line at strix_quick_gate.sh:3137-3141. Also measured and added: - The created_at/started_at gap is not the ~90 min the section quoted. On 2026-09-04 co#983's noema-review waited 7h32m (created 02:15:17Z, started 09:47:37Z) and its strix job 7h11m. Take the gap per run. - served_model=unknown is its own shape: co#983's noema job ended "HTTP Error 502 ... duration=2807.9s, phase=connecting, served_model=unknown". phase=connecting means the call never reached model selection, so the annotation declines to name a candidate instead of guessing one. - A green preflight does not predict the review call. Ten minutes before that 502 the same job reported probed_count 12, ready_count 6 and a route with status "ready" / finish_reason "stop"; the call then ran 46.8 minutes and died at connect. The layers fail independently. - One provider family, two classes: gemma-3-12b-it and gemma-3-4b-it were rejected with http_status 404 while gemma-4-31b-it timed out at ~90s. - Third occupancy sample: co#983's strix job held a runner 319.4 min, and inside it one attempt logged "failed ... after 15357s" (256.0 min) beside attempts of 205s, 176s and 1806s. Nothing is bounded near 1800s. - A partially cancelled head is not green: co#972 held four cancelled required contexts beside 20 successes. Cancelled is missing evidence. - opencode-review success is not evidence of a verdict. .github#1994's job 101585826029 passed in six seconds on the draft short-circuit with zero reviews on the PR; both of 2026-09-06's only two successes were this path. Count reviews by opencode-agent, not workflow-run conclusions. - New section 13, the counter-example: co#971's "DID NOT RAISE" failures are the PR's own source defect -- neither node id exists on main at 414f2297 -- so not every red check in this catalogue is infrastructure. Gate on the merged base: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 118 +++++++++++++++++- 1 file changed, 114 insertions(+), 4 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index e3df15385e..f9b291fba5 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -86,6 +86,18 @@ A merged fix reaches only runs created after the merge. Where a report carries a it in preference to the timestamp: a pre-`#1957` preflight has no `postponed_probed_count` key and stops at `probed_count 6`; a post-`#1957` one carries the key and spends the full 16-probe budget. +**The 90-minute gap is a sample, not a constant — take it from the run you are triaging.** On +2026-09-04 the same PR's two required jobs waited far longer: `contextual-orchestrator#983`'s +`noema-review` (`100887199781`) was created 02:15:17Z and started **09:47:37Z** (7 h 32 m), and its +`strix` job (`100891196281`) was created 02:36:16Z and started **09:47:14Z** (7 h 11 m). At that +spread a fix merged mid-morning is still absent from a run that starts after lunch. + +**A second worked example, from the other side.** `#1953` (name the Strix sandbox bootstrap failure) +merged at **2026-09-06T07:43:39Z**, so its verdict token cannot appear in any run created before that +instant. A pre-`#1953` artifact whose every attempt died in the sandbox therefore still carries the +bare gateway verdict — worked through in §3. The absent marker dates the run; it is not evidence +about the failure. + --- ## 1. `opencode-review` fails with "No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head" @@ -532,8 +544,15 @@ three combinations: `#1938`'s run `34035203943` carries the line *and* the `STRIX_SANDBOX_UNAVAILABLE` verdict with a healthy `ready 6` preflight and `Cost $0.0000 · Tokens 0` (true sandbox class); `#1916`'s run `34027314569` carries the line but its terminal verdict is `orchestrator/free exhausted` (mixed — count it as gateway); `#1946`'s run `34027492927` carries no -such line at all (gateway). **Classify on the `STRIX_PROVIDER_UNAVAILABLE:` / `STRIX_SANDBOX_UNAVAILABLE:` -verdict line, never on a grep.** A fourth specimen closed the set the same day: `#1930`'s run +such line at all (gateway). **Classify on the verdict line, never on a grep — and test it as a +substring.** The two tokens are not alternatives: on post-`#1953` `main` the sandbox verdict is a +*composite* line beginning `STRIX_PROVIDER_UNAVAILABLE: STRIX_SANDBOX_UNAVAILABLE: the last Strix +attempt ended in the sandbox bootstrap …` (`strix_quick_gate.sh:4393`), the leading provider token +kept deliberately so the workflow still classifies a finding-free sandbox outage as incomplete +infrastructure evidence. A test of the form "the line starts with `STRIX_PROVIDER_UNAVAILABLE` → +gateway" therefore misfiles **every** sandbox run. Search for `STRIX_SANDBOX_UNAVAILABLE` as a +substring anywhere in the line, which is what the one consumer that reads it correctly does +(`opencode-review-dispatch.yml:5648`). A fourth specimen closed the set the same day: `#1930`'s run `34027404208` carries `loginAsGuest` in `strix-pr-scope-jibt1j_66a8/strix.log` yet ends `STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted`, with a healthy `ready 6 / probed 16 / escalations_used 2` preflight — a grep files it as a sandbox outage, the @@ -543,8 +562,44 @@ generic check annotation is generic *by design*: `strix_quick_gate.sh:4386` deliberately keeps that token while appending the discriminator, so the annotation alone never tells you which class you have. -**Do not.** Do not blame `served_model`; it is merely the last candidate that failed -(`orchestrator.py:7812`). Do not diagnose it as a provider hang specifically — a 502 here equally +**On a run created before `#1953` merged, the verdict line cannot answer the question at all — use +the retry-reason line.** `contextual-orchestrator#983`'s `strix` job (`100891196281`, created +2026-09-04T02:36:16Z) is the fifth specimen and the cleanest. All three attempts died in +`RuntimeError: loginAsGuest failed after 10 attempts: curl exit 7: … Failed to connect to 127.0.0.1 +port 48080`, the loop announced `Retrying model 'orchestrator/free' due to Caido sandbox bootstrap +timing (attempt 3/3)`, and the terminal verdict was nonetheless the bare +`STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted`. That is not a +misreading of the artifact — it is the corruption `#1953` was written to end, and that commit's own +source comment records the same census damage ("two of six recent Strix artifacts were this class, +with the sidecar reporting ready routes that were never called"). Mechanism: +`has_detected_infrastructure_error()` (`strix_quick_gate.sh:3364-3416` on `main`) counts the Caido +predicate (`:3401`) as one of its members, and the pre-`#1953` emit site collapsed every member to +the one gateway sentence. For such a run the faithful signal is the retry-reason line the retry loop +prints at `:3137-3141`, which names the predicate that actually matched. + +**`served_model=unknown` is a distinct — and more honest — shape than a named `served_model`.** +`contextual-orchestrator#983`'s `noema-review` (`100887199781`) ended `HTTP Error 502: Bad Gateway; +caller attempts=1, duration=2807.9s, phase=connecting, served_model=unknown`. Read `phase=connecting` +first: the call never reached model selection, so there is no candidate to name and the annotation +says so rather than guessing one. Treat `phase=connecting` + `served_model=unknown` as a +sidecar→gateway transport failure and do not go looking for a bad model. + +**A green preflight does not predict the review call.** Ten minutes before that 502 the same job's +preflight reported `probed_count 12`, `ready_count 6`, and a route with `status: "ready"` and +`finish_reason: "stop"` (`nvidia_nim_deepseek_ai_deepseek_v4_pro_0813`). The review call then ran +46.8 minutes and died at connect. The two layers fail independently — preflight measures whether a +model answers, the 502 measures whether the sidecar can reach the gateway at all — so "the preflight +was healthy" is not evidence that the gateway was. (That report carries no `postponed_probed_count` +key, dating it pre-`#1957`: a third independent confirmation of §0's version marker.) + +**One provider and one model family can hold two different classes at once.** In that same preflight +`google/gemma-3-12b-it` and `google/gemma-3-4b-it` were `rejected` with `http_status: 404` — those +models are simply not served — while `google/gemma-4-31b-it` on both NVIDIA keys was `rejected` with +`TimeoutError` at ~90 s. A 404 wants the catalog corrected; a 90 s timeout wants the breaker +discussion above. Collapsing them into "NVIDIA NIM is down" loses both fixes. + +**Do not.** Do not blame `served_model` when it names a model; it is merely the last candidate that +failed (`orchestrator.py:7812`). Do not diagnose it as a provider hang specifically — a 502 here equally covers upstream 5xx, TLS, DNS and connection failures. Do not add `timeout-minutes` to `.github/workflows/noema-review.yml`; its absence is deliberate. Do not edit the PR under review. @@ -748,6 +803,14 @@ five PR scans were left alone. Note what freeing a slot buys while the pool read queued review fails in five minutes with an artifact (signature 3's fail-fast shape) instead of holding a runner for hours without one — still the right outcome. +**Strix job occupancy, three samples — this is execution, not queue wait.** `#1930`'s `strix` job +held its runner **351.8 min**, `#1938`'s **323.9 min**, and `contextual-orchestrator#983`'s +(`100891196281`) **319.4 min** (09:47:14Z→15:06:35Z), while sibling jobs in the same runs finished in +0.1–0.2 min. Inside that last job a **single attempt** logged `Strix run failed for model +'orchestrator/free' after 15357s` (256.0 min), alongside attempts of 205 s, 176 s and 1806 s in the +same job. Any reading that places an upstream ceiling near 1800 s is retired by that one line: there +is no bound there, and one attempt can consume most of the 6-hour Actions maximum by itself. + **Do.** Read each check's actual conclusion. Truly `queued` → leave it and work elsewhere. A workflow that was never assigned a runner → escalate on `.github` #712, #1531, #1219. If a PR of yours has never completed a review cycle, count your own pushes to it before blaming the queue: batch your @@ -928,6 +991,14 @@ auto-updated branch that run may already have been cancelled by a newer head, an never produce a conclusion. Do not respond by merging `main` in yourself — a second updater does not help, and the branch is already being updated more often than the queue can absorb. +**The mirror-image trap: a *partially* cancelled head.** Zero `failure` conclusions does not make a +head green when required contexts are missing. On `contextual-orchestrator#972` (2026-09-07) four +exact-head contexts read `cancelled` — `noema-review` `100796666132`, `strix` `100786422364`, +`coverage-source-tree` `100785092808`, `publish-manual-pr-evidence-status` `100934396825` — beside +**20** `success`, 5 `skipped` and 2 `neutral` runs, and the PR was held. A cancelled required context is *missing* evidence, not +passing evidence, and a larger count of unrelated green checks cannot stand in for it. Count required +contexts and their conclusions; never argue from "20 checks are green". + **The converse trap.** `conclusion == "success"` is not a safe filter for "did this run act". While chasing an unexplained branch update on 2026-09-05, a peer found a run that had acquired a runner and pushed an update-branch three seconds before the cancel reached it; it reports `cancelled`, and a @@ -951,6 +1022,18 @@ requests as drafts by default, so the mismatch is systemic, not a one-off: measu 33 of 138 open `.github` PRs, **16 of 16** open `noema` PRs, and 17 of 64 open `contextual-orchestrator` PRs were drafts — each un-approvable until someone flips it, however green its checks. +**The CI-side tell, and why it corrupts any governed-path KPI.** A draft PR does not fail +`opencode-review`; it **passes** it in about six seconds. Measured on `.github#1994` (run +`34066504222`, job `101585826029`): four steps, 01:13:59Z→01:14:05Z, conclusion `success`, log line +*"PR is still a draft on the live exact head; a current-head OpenCode verdict is not required until +it is marked ready for review."*, with `PR_DRAFT: true` in the step env. The PR carried **zero** +reviews from anyone. Both of the only two `opencode-review` `success` conclusions in `.github` on +2026-09-06 (`#1994`, `#1861`) were this path. So a `success` conclusion on `opencode-review` is not +evidence that a verdict exists: the same conclusion covers a real verdict, this draft short-circuit, +and the skip paths. **Count reviews authored by `opencode-agent` on the exact head, never workflow-run +conclusions** — the two differ by exactly the draft population measured above, which is a third of +open `.github` PRs and was all of `noema`'s. + **Do.** Convert your own PR to ready-for-review the moment its local gates pass. The MCP `update_pull_request` tool takes `draft: false`; REST `PATCH /pulls/{n}` cannot change draft state (it needs GraphQL `markPullRequestReadyForReview`, which the MCP tool wraps). The `ready_for_review` @@ -1017,6 +1100,33 @@ supposed to establish. Do not re-run the job: the same source produces the same --- +## 13. The counter-example: a red check that really is the PR's own source failure + +**Symptom.** A test job ends `2 failed, 3490 passed, 2 skipped in 749.54s` with named assertions — +not a transport error, not a verdict token, not a cancellation. + +**Every other signature in this catalogue is infrastructure, which makes this one easy to +misfile.** Measured on `contextual-orchestrator#971` (job `101589111271`, head `eeb9cc1b`): `FAILED +tests/test_model_discovery_boundaries.py::test_bootstrap_selection_fails_closed_at_unpriced_boundary +- Failed: DID NOT RAISE ValueError` and `FAILED +tests/test_provider_bootstrap.py::test_diverse_selection_fails_closed_at_unpriced_boundary - Failed: +DID NOT RAISE ProviderBootstrapError`. Both are fail-closed boundary tests the PR itself introduces: +neither node id exists on `main` at `414f2297` (`pytest` reports `not found: … (no match in any of +[])` there). The PR's own new tests against the PR's own new code, therefore terminal, +deterministic, and reproducible locally in seconds. + +**The discriminators, in order of cost.** A named assertion or `DID NOT RAISE` in the `short test +summary info` block is source, full stop — infrastructure signatures never reach the assertion phase. +A pass count in the thousands means the suite ran to completion rather than dying in setup. And the +decisive check is cheap: try the failing node id on `origin/main`. It fails there too → not this +PR's (§7's porting rule applies). It does not *exist* there → the PR introduced it, and no amount of +re-running or queue analysis will change the result. + +**Do not.** Do not re-run it, do not attribute it to the queue, the gateway, or the sandbox, and do +not spend a sanctioned re-run on it. + +--- + ## Absence of data flow is not evidence that an edge is safe to cut Under queue saturation every serial `needs:` hop costs a full queue round (measured at 1.89–2.76 h From 7a40831315e3db905f653bdcf8ae699bb7bd8192 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 02:24:59 +0000 Subject: [PATCH 38/50] docs(triage): the created_at rule applies to your own aggregates Recorded against myself. I classified the 20 most recent opencode-review-dispatch failures at 02:20Z, found 15 rejected at the actor gate, and reported that mismatch as the live cause -- two hours after OPENCODE_REPOSITORY_DISPATCH_ACTOR had already been changed at 00:14:14Z and the gate verified open. Every actor-gate rejection in the sample was created before that change. A "most recent N failures" list is dominated by the pre-fix population for as long as the queue is deep, so it keeps reporting a fixed cause as current. Splitting the same 100 runs on the cutoff shows it: 81 of 86 created before it failed; of the 14 created after, 10 were still queued, 1 pending, 1 cancelled, and the only 2 failures were the target allowlist correctly rejecting sibling repositories. So: split a batch on the created_at of the most recent relevant merge or configuration change before concluding anything, and report the halves separately. Signature 1 also carried the actor mismatch as an open condition. It now records the resolution and, more usefully, how to confirm it: read the gate's own "Authorized repository_dispatch actor=... target=..." print from an execution, never the variable value or its updated_at -- this repository has a precedent (2026-09-06T07:01:34Z) of that variable being touched without the change taking effect. Confirmed on run 34069437294, job 101583929799. That run still failed one gate later on "metadata does not match the live pull request: head_sha", so an Authorized line is not evidence that a verdict was published. Refs #1929, #1932, #712. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index f9b291fba5..35a3d5fcb4 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -98,6 +98,19 @@ instant. A pre-`#1953` artifact whose every attempt died in the sandbox therefor bare gateway verdict — worked through in §3. The absent marker dates the run; it is not evidence about the failure. +**The rule applies to your own aggregate measurements, and that is the easier way to get caught.** +A "most recent N failures" sample is dominated by the pre-fix population for as long as the queue is +deep, so it will keep reporting a fixed cause as live. Measured on myself, 2026-09-07T02:20Z: I +classified the 20 most recent `opencode-review-dispatch` failures, found 15 rejected at the actor +gate, and reported the mismatch as the current cause — two hours after the owner had already changed +`OPENCODE_REPOSITORY_DISPATCH_ACTOR` at **00:14:14Z** and verified the gate open. Every actor-gate +rejection in the sample was created before that change. Splitting the same 100 runs on the cutoff +says it plainly: 81 of 86 created before it failed, while of the 14 created after it, 10 were still +queued, 1 pending, 1 cancelled, and the only 2 failures were the *target* allowlist correctly +rejecting sibling repositories. **Do.** Before drawing any conclusion from a batch of runs, split it +on the `created_at` of the most recent relevant merge or configuration change, and report the halves +separately. A lifetime success rate is history; it is not a statement about the current state. + --- ## 1. `opencode-review` fails with "No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head" @@ -136,6 +149,17 @@ updating the variable; neither is a per-PR action. The same variable gates flood is visible from any repository's queue: 01:30–04:30Z on 2026-09-06, 231 `CodeQL Scan Dispatch` runs, two or three per SHA, every one rejected in 3–6 s after ≈65 minutes of queue (signature 7). +**Resolved 2026-09-07T00:14:14Z — and here is how to check that for yourself.** The variable now +reads `github-actions[bot],opencode-agent[bot]`, so the gate admits the App identity. Verify it from +an *execution*, never from the variable's value or its `updated_at`: this repository has a precedent +(2026-09-06T07:01:34Z) of the same variable being touched without the change taking effect. The +line to look for is the gate's own success print, `Authorized repository_dispatch actor=… sender=… +target=…` — confirmed on run `34069437294`, job `101583929799`, whose env block shows the +comma-separated allowlist. Note the run still failed, one gate later, on `metadata does not match the +live pull request: head_sha` (supplied `f8be5e31`, live `6e1194aa`): reopening the actor gate exposes +the next check rather than producing a verdict, so do not read an `Authorized …` line as evidence +that a verdict was published. + **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. From ec033bd0bb844cb02a1ce28904c3b3058c8f0679 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 03:16:47 +0000 Subject: [PATCH 39/50] docs(triage): record the in-place recovery for gate-killed runs A run that died only at the dispatch actor gate does not need a push to recover. rerun-failed-jobs re-executes just the failed jobs against the now-open gate, bumping run_attempt while leaving the head and its exact-head evidence intact. Verified on #1946: run 34027493446 still reports head 790ef33ea60a at run_attempt 2, with 101610595401 and 101610595600 queued at steps 0. Signature 1 already says not to reflexively re-run the failed job, so the entry states why this is not that: the rule forbids re-running unchanged inputs hoping for a different result, and here the allowlist actually changed. It also notes the ordering -- confirm the gate is open by the "Authorized repository_dispatch" line before rerunning, or the rerun just reproduces the rejection -- and why a rerun beats a push, which would reset the head and throw away the evidence the run holds. Refs #1946, #1929. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 35a3d5fcb4..4c38532344 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -160,6 +160,18 @@ live pull request: head_sha` (supplied `f8be5e31`, live `6e1194aa`): reopening t the next check rather than producing a verdict, so do not read an `Authorized …` line as evidence that a verdict was published. +**Recovering the runs the closed gate already killed — without a push.** A run that died only at the +actor gate is recoverable in place: `POST /actions/runs/{id}/rerun-failed-jobs` re-executes just the +failed jobs against the *now-open* gate, bumping `run_attempt` while leaving the head and every +piece of exact-head evidence untouched. Verified on `#1946`: run `34027493446` still reports head +`790ef33ea60a` at `run_attempt: 2`, with `101610595401` (actions) and `101610595600` (python) queued +at `steps 0` and `Detect CodeQL languages` already `success`. This does **not** contradict the "do +not reflexively re-run" rule two paragraphs up — that rule forbids re-running in the hope of a +different result from unchanged inputs. Here an input genuinely changed (the allowlist), so the +rerun is the cheapest correct recovery, and it is strictly better than pushing: a push would reset +the head and discard the evidence the run already holds. Establish that the gate is open *first*, by +the `Authorized …` line, or the rerun just reproduces the rejection. + **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. From 1a7cafd288fd6864c56fb7fa7ab7e8116abaa216 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 03:37:48 +0000 Subject: [PATCH 40/50] docs(doctoring): mark the org-queue-sweep rotation record historical The document still read as current operations -- a 15-minute sweep tick, a persistent rotation counter, ORG_SWEEP_REVIEW_DISPATCH_LIMIT budgets and a rate-limit path sleeping up to 60 seconds -- for a job that no longer exists. Verified against protected main rather than taken on report: #1878 ("fix(actions): remove organization queue sweep") merged as 1b65dbc35e7183722ad77894e2d80b39993be90d, the later lineage adds 85f4ef926feed08cf502a8e6bf26e495a0bb7bed ("ci(actions): fold head coalescing into scheduler"), both are ancestors of main@c9052e607e5f3cc76e73207e7786b21500721b79, and no workflow there defines the job. The two remaining mentions in the tree are a code comment and test_strix_quick_gate.sh:1565's negative assertion, which pins the absence rather than describing a live job. The RCA and evidence are kept verbatim; only their claim to describe the present is retired. The notice says explicitly not to restore the sweep or build a replacement from the document, and states that it makes no claim about queue health, runner admission, or model review, which stay open on #712, #1531 and #1219. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Refs #1878, #1796. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/org-queue-sweep-rotation.md | 25 ++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/doctoring/org-queue-sweep-rotation.md b/docs/doctoring/org-queue-sweep-rotation.md index 03784d0b7e..3713a2bb10 100644 --- a/docs/doctoring/org-queue-sweep-rotation.md +++ b/docs/doctoring/org-queue-sweep-rotation.md @@ -1,5 +1,30 @@ # Org-queue-sweep review-dispatch rotation +> **HISTORICAL — SUPERSEDED. The subject of this document no longer exists.** +> +> `org-queue-sweep` was removed from `pr-review-merge-scheduler.yml` by +> [#1878](https://github.com/ContextualWisdomLab/.github/pull/1878) +> ("fix(actions): remove organization queue sweep"), merged as +> `1b65dbc35e7183722ad77894e2d80b39993be90d`. Later protected lineage adds +> `85f4ef926feed08cf502a8e6bf26e495a0bb7bed` ("ci(actions): fold head coalescing +> into scheduler", 2026-09-05). Both are ancestors of protected +> `main@c9052e607e5f3cc76e73207e7786b21500721b79`, whose scheduler tree contains +> no such job; `scripts/ci/test_strix_quick_gate.sh:1565` now pins its absence +> (`assert_file_not_contains … "scheduler does not consume a runner on +> organization-wide polling"`). +> +> **Everything below is preserved as the RCA and evidence record, and nothing in +> it describes current behaviour.** The 15-minute tick, the persistent rotation +> counter, the `ORG_SWEEP_REVIEW_DISPATCH_LIMIT` / +> `ORG_SWEEP_STACKED_REVIEW_DISPATCH_LIMIT` budgets and the rate-limit path with +> its up-to-60-second sleep are all descriptions of the removed job. Do not +> restore the sweep and do not build a replacement sweep from this document. +> +> Current authority for the scheduler is #1878 and the coalescing lineage above. +> This notice retires one document's claim to describe the present; it makes no +> statement about queue health, runner admission, or model review, which remain +> independently open (`#712`, `#1531`, `#1219`). + ## Problem `org-queue-sweep` in `pr-review-merge-scheduler.yml` walks every organization From f1226b934865cb5b0f18753c9297d8e3e6c4841c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 03:39:11 +0000 Subject: [PATCH 41/50] docs(triage): record the first post-gate signature 9 measurement Once the actor allowlist reopened at 2026-09-07T00:14:14Z, the same red CodeQL check stops meaning signature 1 and starts meaning signature 9, so the DISPATCH_OUTCOME discriminator is what separates them. First measurement on a run created after that instant: contextual-orchestrator#971's run 34072366652 (created 01:14:10Z) failed all three compatibility jobs -- 101600579562, 101600579590, 101600579614 -- with DISPATCH_OUTCOME success and VERDICT_STATE pending. The dispatch was authorized; only the child scan had not finished. The repository owner independently read the same shape on #1946 as the intentional first-pass runner-release protocol. Recorded because it inverts the naive reading: this failure on a post-fix run is evidence the authorization path works, not evidence against it, and the in-place rerun-failed-jobs recovery collects the verdict without moving the head. No action was taken on #971 itself -- an owner-driven Draft whose body tracks this head under an explicit do-not-disturb. Refs #1929, #1946. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 4c38532344..c91b6c2f24 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -990,6 +990,19 @@ failure as a defect in the PR's code — the job never got as far as analysing i signature 1 without checking `DISPATCH_OUTCOME` — the two produce the same red check and take different actions. +**Post-`#1929` this is the shape you should expect, and it is the discriminator earning its keep.** +Once the actor allowlist reopened (2026-09-07T00:14:14Z), the same red check stops meaning signature +1 and starts meaning this. First measurement on a run created after that instant: +`contextual-orchestrator#971`'s run `34072366652` (created 01:14:10Z) failed all three CodeQL +compatibility jobs — `101600579562` actions, `101600579590` python, `101600579614` +javascript-typescript — with `DISPATCH_OUTCOME: success` and `VERDICT_STATE: pending`. The dispatch +was *authorized*; only the child scan had not finished. The repository owner independently described +the same shape on `#1946` as "the intentional first-pass runner-release protocol, not an analysis +failure", which is the right reading. So a `DISPATCH_OUTCOME: success` / `VERDICT_STATE: pending` +failure on a post-fix run is evidence the authorization path is working, not evidence against it — +and §1's in-place `rerun-failed-jobs` recovery is what collects the verdict once the child scan +lands, without moving the head. + --- ## 10. Every check on the head reads `cancelled`, and the PR never completes a check cycle From 63d52b3ecfeaab1fe08c220ec01c1bc7288ffb96 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:08:47 +0000 Subject: [PATCH 42/50] docs(triage): guard the rerun recovery against the CodeQL attempt-1 rule The recovery entry two commits back told a reader to collect a pending verdict with rerun-failed-jobs. On a CodeQL compatibility job that advice can make things worse, so it now carries the precondition. codeql-pr.yml:195-221 looks for an authenticated terminal codeql-dispatch/ commit status created by opencode-agent, and when RUN_ATTEMPT != 1 finds none it fails the job outright with "Exact CodeQL job was rerun without an authenticated terminal verdict". It does not re-dispatch: that path, REQUIRED_RUN_ID/REQUIRED_JOB_ID validation included, is reachable only on attempt 1. So rerunning while the verdict is still pending -- signature 9's shape, which is exactly when a rerun looks attractive -- turns a recoverable pending into a terminal failure and spends the attempt that could have re-dispatched. Both entries now state the ordering, with a gh api + jq check for the status before rerunning: terminal state from opencode-agent[bot] means rerun, anything else means wait for the child scan. Observed on fast-mlsirm#1692 leaf attempt 2 job 101562653014, with .github#1902 named as the repair owner. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Refs #1902. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index c91b6c2f24..936195904e 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -172,6 +172,26 @@ rerun is the cheapest correct recovery, and it is strictly better than pushing: the head and discard the evidence the run already holds. Establish that the gate is open *first*, by the `Authorized …` line, or the rerun just reproduces the rejection. +**On a CodeQL compatibility job the rerun has a hard precondition, and getting it wrong is worse than +doing nothing.** `codeql-pr.yml:195-221` looks for an authenticated terminal +`codeql-dispatch/` commit status — state `success`, `failure` or `error`, created by +`opencode-agent` or `opencode-agent[bot]` — and when `RUN_ATTEMPT != 1` finds none, it fails the job +outright with `::error::Exact CodeQL job was rerun without an authenticated terminal verdict.` It does +**not** re-dispatch: the whole re-dispatch path, `REQUIRED_RUN_ID`/`REQUIRED_JOB_ID` validation +included, is reachable only on attempt 1. So rerunning while the verdict is still `pending` — §9's +shape, the very state that makes a rerun tempting — converts a recoverable pending into a terminal +failure *and* spends the attempt that could have re-dispatched. Observed on `fast-mlsirm#1692`, leaf +attempt 2 job `101562653014`, with `.github#1902` named as the repair owner. **Do.** Confirm the +authenticated status exists on the exact head before rerunning: + +```bash +gh api "repos///commits//statuses" | + jq -r '.[] | select(.context=="codeql-dispatch/") | "\(.state) \(.creator.login)"' +``` + +Terminal state from `opencode-agent[bot]` → rerun. Anything else → leave it and wait for the child +scan; there is nothing a rerun can collect yet. + **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. @@ -1001,7 +1021,9 @@ the same shape on `#1946` as "the intentional first-pass runner-release protocol failure", which is the right reading. So a `DISPATCH_OUTCOME: success` / `VERDICT_STATE: pending` failure on a post-fix run is evidence the authorization path is working, not evidence against it — and §1's in-place `rerun-failed-jobs` recovery is what collects the verdict once the child scan -lands, without moving the head. +lands, without moving the head. Read §1's precondition before doing that here: while +`VERDICT_STATE` is still `pending` there is no authenticated status to collect, and a rerun in that +state fails the job terminally instead of recovering it. --- From 02ab27395eac0bfce63010b8f5804c7f0b5b818c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:21:29 +0000 Subject: [PATCH 43/50] docs(triage): record what the reopened gate exposed, as a hypothesis Four hours after the allowlist opened, 31 opencode-review-dispatch runs had been created and none had succeeded: 25 queued, 1 pending, 2 cancelled, 3 failed. None of the three failures is the actor gate -- two are the target allowlist correctly refusing sibling repositories, and 34077699386 is "metadata does not match the live pull request: base_ref,base_sha". With 34069437294's head_sha mismatch that is two instances of one shape: metadata accurate when sent, stale by the time validate-pr-metadata ran. That suggests a structural interaction with signature 7 -- the longer a dispatch waits for a runner, the likelier the PR moved underneath it -- and predicts that metadata-mismatch rejections track queue latency and concentrate on fast-moving heads. Two instances do not establish it, so it is written as a hypothesis carrying its own evidence rather than as a measurement, and the solid part is stated separately: post-fix, a dispatch failure is no longer evidence about the actor gate, and throughput is now a queue question rather than an authorization one. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 936195904e..49eada9931 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -192,6 +192,24 @@ gh api "repos///commits//statuses" | Terminal state from `opencode-agent[bot]` → rerun. Anything else → leave it and wait for the child scan; there is nothing a rerun can collect yet. +**What the reopened gate exposed next, offered as a hypothesis rather than a finding.** Four hours +after the allowlist opened, 31 `opencode-review-dispatch` runs had been created and **none had +succeeded**: 25 still queued, 1 pending, 2 cancelled, and 3 failed. Not one of the three is the actor +gate. Two are the *target* allowlist correctly refusing sibling repositories (`seedream_evasepic`, +`argos`); the third, `34077699386`, is `metadata does not match the live pull request: +base_ref,base_sha`. Add `34069437294`'s `head_sha` mismatch and there are two instances of the same +shape: the dispatch carried metadata that was accurate when it was sent and stale by the time +`validate-pr-metadata` ran. + +That suggests a structural interaction between signature 7 and this validator — the longer a +dispatch waits for a runner, the likelier the PR has moved underneath it — and it yields a testable +prediction: metadata-mismatch rejections should rise and fall with queue latency, and should +concentrate on PRs whose heads move often. **Two instances do not establish that**, and it is +recorded here as a hypothesis with its evidence so the next person can test it rather than rediscover +it. What can be said without the hypothesis is narrower and solid: post-fix, a dispatch failure is no +longer evidence about the actor gate, and the throughput question is now queue depth, not +authorization. + **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. From 9f0d1d00089504ef8ffe7c19b7f52ba5dc8907f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:29:26 +0000 Subject: [PATCH 44/50] docs(triage): #1946's rerun failed, so it is the counter-example now An earlier commit in this batch cited #1946's run 34027493446 as verification that rerun-failed-jobs recovers a gate-killed run. At 02:40Z that looked right: run_attempt 2 on the same head, both CodeQL jobs queued at steps 0, Detect CodeQL languages already success. At 04:26Z the actions job 101610595401 ended with RUN_ATTEMPT 2 and "Exact CodeQL job was rerun without an authenticated terminal verdict." So the run I held up as a working recovery is an instance of the anti-pattern the guard commit describes, and it would have shipped as a worked example telling readers to do the thing that just failed. The entry now leads with the general case, marks the CodeQL job as the exception, and walks #1946 through as the counter-example -- what it looked like mid-flight and how it ended -- because the mid-flight appearance is precisely what makes the mistake tempting. Nothing was pushed with the wrong version. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Refs #1946, #1902. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 33 ++++++++++++------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 49eada9931..702951b52b 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -160,17 +160,28 @@ live pull request: head_sha` (supplied `f8be5e31`, live `6e1194aa`): reopening t the next check rather than producing a verdict, so do not read an `Authorized …` line as evidence that a verdict was published. -**Recovering the runs the closed gate already killed — without a push.** A run that died only at the -actor gate is recoverable in place: `POST /actions/runs/{id}/rerun-failed-jobs` re-executes just the -failed jobs against the *now-open* gate, bumping `run_attempt` while leaving the head and every -piece of exact-head evidence untouched. Verified on `#1946`: run `34027493446` still reports head -`790ef33ea60a` at `run_attempt: 2`, with `101610595401` (actions) and `101610595600` (python) queued -at `steps 0` and `Detect CodeQL languages` already `success`. This does **not** contradict the "do -not reflexively re-run" rule two paragraphs up — that rule forbids re-running in the hope of a -different result from unchanged inputs. Here an input genuinely changed (the allowlist), so the -rerun is the cheapest correct recovery, and it is strictly better than pushing: a push would reset -the head and discard the evidence the run already holds. Establish that the gate is open *first*, by -the `Authorized …` line, or the rerun just reproduces the rejection. +**Recovering the runs the closed gate already killed — without a push, and only where the job allows +it.** In principle a run that died only at the actor gate is recoverable in place: +`POST /actions/runs/{id}/rerun-failed-jobs` re-executes just the failed jobs against the *now-open* +gate, bumping `run_attempt` while leaving the head and every piece of exact-head evidence untouched. +That is strictly better than pushing, which would reset the head and discard the evidence the run +already holds, and it does not contradict the "do not reflexively re-run" rule two paragraphs up: +that rule forbids re-running unchanged inputs hoping for a different result, and here an input +genuinely changed. Establish that the gate is open *first*, by the `Authorized …` line, or the rerun +only reproduces the rejection. + +**On a CodeQL compatibility job it does not work, and `#1946` is the worked counter-example — watch +it fail.** Run `34027493446` was rerun on the same head `790ef33ea60a`; at 02:40Z it looked like a +clean recovery, `run_attempt: 2` with `101610595401` (actions) and `101610595600` (python) queued at +`steps 0` and `Detect CodeQL languages` already `success`. At 04:26Z the actions job ended: + +``` +RUN_ATTEMPT: 2 +##[error]Exact CodeQL job was rerun without an authenticated terminal verdict. +``` + +The rerun did not collect a verdict; it destroyed the run's remaining chance of one. Read the next +paragraph before reaching for `rerun-failed-jobs` on any CodeQL job. **On a CodeQL compatibility job the rerun has a hard precondition, and getting it wrong is worse than doing nothing.** `codeql-pr.yml:195-221` looks for an authenticated terminal From 36761283f00ca07f7b5468db8a038a78d8197b1c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:32:56 +0000 Subject: [PATCH 45/50] docs(triage): ground the concurrency guidance in the documented contract Section 7 told readers to keep cancel-in-progress: true and not switch it to false, with a queue-cost argument. The conclusion was right but the reasoning left the common misconception standing: that false is how you preserve a pending run. Re-verified against GitHub's "Control the concurrency of workflows and jobs" rather than restated from the request: false protects a running job only, and "any existing pending job or workflow in the same concurrency group will be canceled and the new queued job or workflow will take its place"; queue: max is what retains pending work, up to 100, cancelling the excess; queue: max with cancel-in-progress: true is "not allowed and will result in a workflow validation error", so the two are alternatives rather than a pair; and FIFO runs "according to the time each one started waiting on the concurrency group, not the time each workflow was dispatched", so queue order is not head, version or tag authority. The lane split now follows from that. PR validation/review/security keep queue: single with trigger-aware cancel-in-progress: true, justified narrowly as coalescing superseded validation evidence for the same PR and workflow. Only destructive or publication lanes group by the actual target and consider queue: max with non-cancelling semantics, and even there FIFO is not the release order. Guidance only. naruon#1586 owns release/deploy ordering and is already applying this, so the entry says not to build a second implementation from it. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 702951b52b..ccb0f06311 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -840,6 +840,34 @@ switch it to `false`: **62% of the cancelled jobs had not been assigned a runner cancellation costs nothing real, and forcing them to run would return all 62% to a queue that is already at the ceiling. +**And `false` would not buy what people think it buys.** Re-verified against GitHub's own +*Control the concurrency of workflows and jobs* on 2026-09-07: + +1. `cancel-in-progress: false` protects a **running** job. It does not preserve a **pending** one — + "any existing `pending` job or workflow in the same concurrency group will be canceled and the new + queued job or workflow will take its place." Writing `cancel-in-progress: false` and claiming + pending preservation is simply wrong, and it is the most common mistake in this area. +2. `queue: max` is what retains pending work: "Up to 100 jobs or workflow runs can be `pending` in + the concurrency group. When the queue is full, any additional jobs or workflow runs are canceled." +3. `queue: max` and `cancel-in-progress: true` **cannot be combined** — "not allowed and will result + in a workflow validation error." So the two levers are alternatives, not a pair to tune. +4. FIFO is not ordering authority: runs are processed "according to the time each one started waiting + on the concurrency group, not the time each workflow was dispatched." Queue order therefore says + nothing about head, version, or tag order. + +**The lane split that follows from it.** PR validation, review and security keep `queue: single` with +trigger-aware `cancel-in-progress: true`, and the justification is narrow: it coalesces *superseded +validation evidence* for the same PR and workflow. That is all it does — it is not a statement about +ordering or about preserving intent. Only a destructive or publication lane, where losing an intent +is not acceptable (release, deploy, migration), should group by the actual target — environment, +registry package, version, migration target — and consider `queue: max` with non-cancelling +semantics. Even there, FIFO is not the release order: require protected head, tag, version, digest, +idempotency and lock verification separately. + +`ContextualWisdomLab/naruon#1586` is the release/deploy ordering owner and is already applying this. +Do not build a second implementation or branch for it from this catalogue; this section is guidance +only. + **What actually holds the slots — measured 2026-09-05T14:27Z.** Occupancy is a property of *jobs*, not runs: list every in-progress run's jobs (`/actions/runs/{id}/jobs`) and read `started_at` and `runner_name`. A run "in progress for 10 hours" turned out to be 8 hours of queue plus 2 hours of slot — From b6ffcc28b2b9ab278c64fba5ec7068f1ec5396d3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:36:15 +0000 Subject: [PATCH 46/50] docs(triage): index the actionlint queue-key schema lag GitHub documents concurrency.queue, but actionlint 1.7.12 rejects a minimal queue: max workflow with unexpected key "queue" (exit 1), so the documented syntax cannot pass the governed validator today. Verified the cause at the source rather than inferring it from the error string: actionlint v1.7.12's Concurrency struct in ast.go models only Group and CancelInProgress, with no queue field, so the key is unknown to it by construction. The entry says so explicitly and attributes the observed error text and exit code to the reproduction reported on #1913, because actionlint is not installed in this container and I did not run it. The entry states that this is validator-schema lag rather than evidence against the feature, and that it is not permission to suppress, downgrade or exempt actionlint. queue: max stays available to a lossless release/deploy/migration lane only once the canonical toolchain can validate it, and only with target locking plus protected head/tag/version/digest verification and idempotency. Toolchain ownership is #1231; naruon#1586 owns release/deploy ordering and did not push the rejected key. Guidance only, no workflow change. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Refs #1231, #1586. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index ccb0f06311..4c55227bbb 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -864,9 +864,24 @@ registry package, version, migration target — and consider `queue: max` with n semantics. Even there, FIFO is not the release order: require protected head, tag, version, digest, idempotency and lock verification separately. -`ContextualWisdomLab/naruon#1586` is the release/deploy ordering owner and is already applying this. -Do not build a second implementation or branch for it from this catalogue; this section is guidance -only. +**Before anyone reaches for `queue: max`: the governed validator does not know the key yet.** +`actionlint 1.7.12` rejects a minimal `concurrency.queue: max` workflow with `unexpected key "queue"` +(exit 1). That is validator-schema lag, not evidence against the feature — confirmed at the source +rather than inferred from the error: actionlint `v1.7.12`'s `Concurrency` struct in `ast.go` models +only `Group` and `CancelInProgress`, with no `queue` field at all, so any `queue` key is unknown to +it by construction. (The struct is what this catalogue verified; the exact error text and exit code +come from the reproduction reported on `#1913`, since actionlint is not installed in this session's +container.) + +**Do not** suppress, downgrade, or exempt actionlint to get such a workflow through — the lint is +reporting its own schema honestly. A lossless release/deploy/migration lane may adopt `queue: max` +only once the canonical toolchain path can validate the documented syntax, and only together with +actual target locking plus protected head, tag, version and digest verification and idempotency. +Toolchain ownership for that is `#1231`. + +`ContextualWisdomLab/naruon#1586` is the release/deploy ordering owner and is already applying the +contract above; it did not push the currently rejected key. Do not build a second implementation or +branch from this catalogue; this section is guidance only. **What actually holds the slots — measured 2026-09-05T14:27Z.** Occupancy is a property of *jobs*, not runs: list every in-progress run's jobs (`/actions/runs/{id}/jobs`) and read `started_at` and From 57432c67dcf34448212eb63bd8de902dbe58fbe2 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 04:46:12 +0000 Subject: [PATCH 47/50] docs: retire the sweep record and scope the occupancy numbers Four related repairs, all documentation and comments; no workflow or census implementation, and no claim that any queue or 60-job definition of done is complete. docs/doctoring/actions-queue-saturation-hourly-sweep.md still read "Status: active repair evidence" with canonical repair #1630 and described the 15-minute-to-hourly sweep change as a live production contract. #1878 (merge 1b65dbc35e7183722ad77894e2d80b39993be90d) deleted the org-queue-sweep source outright and 85f4ef926feed08cf502a8e6bf26e495a0bb7bed folded head coalescing into the scheduler; both are ancestors of main@c9052e607 and no such job remains. It is now marked HISTORICAL / SUPERSEDED with those pointers, RCA kept intact. audit_org_codeql_coverage.py cited that stale document as current evidence for real queue congestion. The 35-day constant is unchanged and its rationale stands; only the pointers are corrected, split explicitly into HISTORICAL records for the removed sweep and #1938's BOUNDED LATER EVIDENCE, with a note that neither proves current org-wide occupancy and that this constant is not derived from one. The triage catalog stated "55 of 60 running jobs" with no method and no scope. It now carries both: a running-slot count is a count of jobs and never of runs, so actions/runs?status=in_progress and queued counts are not occupancy numerators; admit a job only when status is in_progress AND it holds a real runner assignment, keeping queued work as separate backlog. The three-repository sample is labelled a bounded precedent rather than an org census, and the shared-ceiling phrasing is bound to its 2026-09-04 reading. Verified independently rather than assumed: no script under scripts/ci/ reads runner_id or runner_name, and audit_org_codeql_coverage.py is a CodeQL coverage census. Org-wide occupancy is recorded as an open collection gap instead of being implied by a sample. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Refs #1878, #1630, #1938, #1231. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- .../actions-queue-saturation-hourly-sweep.md | 16 ++++++++-- docs/doctoring/ci-failure-signature-triage.md | 29 ++++++++++++++++--- scripts/ci/audit_org_codeql_coverage.py | 21 +++++++++++--- 3 files changed, 56 insertions(+), 10 deletions(-) diff --git a/docs/doctoring/actions-queue-saturation-hourly-sweep.md b/docs/doctoring/actions-queue-saturation-hourly-sweep.md index c68f91d34c..19a2804f7b 100644 --- a/docs/doctoring/actions-queue-saturation-hourly-sweep.md +++ b/docs/doctoring/actions-queue-saturation-hourly-sweep.md @@ -1,8 +1,20 @@ # Actions queue saturation: hourly organization sweep -**Status:** active repair evidence +**Status:** HISTORICAL / SUPERSEDED — preserved as RCA and repair evidence; not current behaviour **Owning repository:** `ContextualWisdomLab/.github` -**Canonical repair PR:** `#1630` +**Canonical repair PR (historical):** `#1630` +**Superseded by:** `#1878`, merged `1b65dbc35e7183722ad77894e2d80b39993be90d`, which removed the +organization-wide `org-queue-sweep` source outright; later lineage adds +`85f4ef926feed08cf502a8e6bf26e495a0bb7bed` ("ci(actions): fold head coalescing into scheduler"). +Both are ancestors of protected `main@c9052e607e5f3cc76e73207e7786b21500721b79`, whose scheduler tree +contains no such job. + +> **Read the rest of this document as history.** The 15-minute admission, the hourly heartbeat that +> replaced it, and the rotation-counter cadence all describe a job that no longer exists. Nothing +> here is a current production contract, no new deletion implementation is required, and this +> document must not be cited as evidence about present queue behaviour. The RCA and the repair record +> are kept intact deliberately. + **Protected baseline:** `main@4ae90e18b03a3a455e13e501628010cabc5c37a8` ## Root cause diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 4c55227bbb..a347212c20 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -808,16 +808,37 @@ in-progress; `contextual-orchestrator` 93; `noema` 53. `scripts/ci/pr_review_merge_scheduler_core.py:342-343` places `QUEUED` in `RUNNING_CHECK_STATES`, kept separate from `FAILED_CHECK_CONCLUSIONS`. Each `needs:` stage queues separately, so end-to-end delay routinely exceeds 8 hours, and some runs queued since 2026-08-19 never started at all. There is -no 4-hour upper bound. The organization-wide ceiling is saturated: 55 of 60 running jobs, with -`.github` holding ~28% of the waiting volume while receiving ~4% of the execution slots. +no 4-hour upper bound. At the 2026-09-04 reading the ceiling was saturated at **55 of 60 running +jobs**, with `.github` holding ~28% of the waiting volume while receiving ~4% of the execution slots. + +**Read that number with its method attached, because the obvious way to compute it is wrong.** +A running-slot count is a count of **jobs**, never of runs: one workflow run holds many jobs, and a +`queued` run or job occupies no runner at all. So `actions/runs?status=in_progress` — and any count of +queued runs — is not an occupancy numerator, however tempting the single call is. To re-measure, +enumerate the *job* collection and admit a job to the numerator only when `status == in_progress` +**and** it carries a real runner assignment (`runner_id` / runner identity); keep queued jobs and +runs as a separate backlog series. The discriminators for a placeholder assignment are in the +occupancy paragraph below (`steps > 0`, `created_at < started_at`). + +**And read it with its scope attached.** That figure and the job-level measurement below came from +`.github`, `noema` and `contextual-orchestrator` — three repositories, a bounded precedent, not an +organization-wide census. `docs/doctoring/actions-queue-saturation-hourly-sweep.md` and +`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` are *historical* RCA and +plan/observation records for the `org-queue-sweep` that `#1878` removed; do not cite either, or these +samples, as proof of current organization-wide occupancy. **There is no org-wide occupancy collector +in the tree today** — a narrow search of protected `main` finds no script in `scripts/ci/` reading +`runner_id`/`runner_name`, and `scripts/ci/audit_org_codeql_coverage.py` is a CodeQL *coverage* +census, not an occupancy collector. Organization-wide occupancy is therefore an open collection gap, +recorded here as one rather than papered over with a sample. Nothing in this section closes a queue +or 60-job definition of done. **The other half of the shortage is occupancy, not just arrival rate — and one job can be enormous.** Measured 2026-09-06 on `#1930`'s run `34027404208`: the `strix` job held a runner from 11:58:00Z to 17:49:47Z, **351.8 minutes**, and ended with no verdict. That is execution, not queue wait — the run's other jobs (`Detect changed scope`, `Admit current pull request head`, `cancel-superseded-pr-runs`) each completed in tenths of a minute at 11:20Z, so this job was assigned -and running for the whole span. Against a 10–30 minute normal scan and a shared 60-job ceiling, one -such job costs roughly a twelfth of the organization's concurrent capacity for a working day. A +and running for the whole span. Against a 10–30 minute normal scan and the 60-job ceiling as read on +2026-09-04, one such job costs roughly a twelfth of that concurrent capacity for a working day. A second sample the same day ran 2 h 46 m (`#1519`, run `34030605428`). When triaging a queue shortage, measure per-job `started_at`→`completed_at` on the longest-running jobs as well as the queued count; capping or bounding those is a different remedy from adding capacity. diff --git a/scripts/ci/audit_org_codeql_coverage.py b/scripts/ci/audit_org_codeql_coverage.py index bdbc835491..9d90d1391c 100644 --- a/scripts/ci/audit_org_codeql_coverage.py +++ b/scripts/ci/audit_org_codeql_coverage.py @@ -34,10 +34,23 @@ # # 35 days is deliberately 5x that observed 7-day interval: a safety margin # against a single missed or delayed scheduled run (a holiday, a GitHub -# platform incident, or this organization's own well-documented Actions -# queue congestion under hosted-runner saturation -- see -# docs/doctoring/actions-queue-saturation-hourly-sweep.md, a real, observed -# risk here, not hypothetical), not an unexplained rule of thumb. +# platform incident, or Actions queue congestion under hosted-runner +# saturation, a risk this organization has observed rather than a +# hypothetical one), not an unexplained rule of thumb. +# +# Evidence pointers, separated by currency because the older records +# describe machinery that no longer exists. HISTORICAL: +# docs/doctoring/actions-queue-saturation-hourly-sweep.md and +# docs/doctoring/actions-plan-concurrency-ceiling-20260903.md are RCA and +# plan/observation records for the removed org-queue-sweep (deleted by +# .github#1878, merge 1b65dbc35e7183722ad77894e2d80b39993be90d); neither +# describes current implementation. BOUNDED LATER EVIDENCE: .github#1938 +# enumerated in-progress runs' JOBS with started_at/runner_name across +# .github, noema and contextual-orchestrator only -- a three-repository +# sample, not an organization-wide census. No pointer here is proof of +# current organization-wide runner occupancy, and this constant is not +# derived from one; it is a freshness margin, and changing it needs its own +# evidence. CODEQL_ANALYSIS_FRESHNESS_DAYS = 35 From 5987f02747e074326910489b4ca100ab8a6a3ef6 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 05:25:28 +0000 Subject: [PATCH 48/50] docs(triage): self-review pass before pushing the batch Two defects found in my own unpushed work by re-checking it rather than trusting it. The occupancy entry cited "a narrow search of protected main finds no script in scripts/ci/ reading runner_id/runner_name". True on main, but the same grep now matches on this branch -- because the evidence comment I added to audit_org_codeql_coverage.py contains the words. A check that your own edit satisfies has stopped being evidence, so the entry now states the fact as "no script reads either field", gives the exact origin/main grep, and names line 48 as prose rather than code. The post-gate snapshot has also moved, and a stale snapshot would have undersold its own conclusion. Five hours after the allowlist opened it is 38 runs, still zero successes, and the failures are 6 rather than 3: four target-allowlist refusals of sibling repositories (seedream_evasepic, argos, g7, 9drive) and two metadata mismatches (34077699386 base_ref/base_sha, 34078693162 head_sha). Zero actor-gate rejections in the larger sample, which is the stronger version of the claim. The stale-metadata hypothesis is now three instances counting 34069437294, and the entry says three do not establish it either. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 28 +++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index a347212c20..426fb3cb3b 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -203,19 +203,20 @@ gh api "repos///commits//statuses" | Terminal state from `opencode-agent[bot]` → rerun. Anything else → leave it and wait for the child scan; there is nothing a rerun can collect yet. -**What the reopened gate exposed next, offered as a hypothesis rather than a finding.** Four hours -after the allowlist opened, 31 `opencode-review-dispatch` runs had been created and **none had -succeeded**: 25 still queued, 1 pending, 2 cancelled, and 3 failed. Not one of the three is the actor -gate. Two are the *target* allowlist correctly refusing sibling repositories (`seedream_evasepic`, -`argos`); the third, `34077699386`, is `metadata does not match the live pull request: -base_ref,base_sha`. Add `34069437294`'s `head_sha` mismatch and there are two instances of the same -shape: the dispatch carried metadata that was accurate when it was sent and stale by the time -`validate-pr-metadata` ran. +**What the reopened gate exposed next, offered as a hypothesis rather than a finding.** Five hours +after the allowlist opened, 38 `opencode-review-dispatch` runs had been created and **none had +succeeded**: 28 still queued, 1 pending, 1 in progress, 2 cancelled, and 6 failed. **Not one of the +six is the actor gate.** Four are the *target* allowlist correctly refusing sibling repositories +(`seedream_evasepic`, `argos`, `g7`, `9drive`) — fail-closed, working as written. The other two are +one shape: `34077699386` (`base_ref,base_sha`) and `34078693162` (`head_sha`), both `metadata does not +match the live pull request`. With `34069437294`'s `head_sha` mismatch on the CodeQL dispatcher that +is three instances of the dispatch carrying metadata that was accurate when sent and stale by the +time `validate-pr-metadata` ran. That suggests a structural interaction between signature 7 and this validator — the longer a dispatch waits for a runner, the likelier the PR has moved underneath it — and it yields a testable prediction: metadata-mismatch rejections should rise and fall with queue latency, and should -concentrate on PRs whose heads move often. **Two instances do not establish that**, and it is +concentrate on PRs whose heads move often. **Three instances do not establish that**, and it is recorded here as a hypothesis with its evidence so the next person can test it rather than rediscover it. What can be said without the hypothesis is narrower and solid: post-fix, a dispatch failure is no longer evidence about the actor gate, and the throughput question is now queue depth, not @@ -826,9 +827,12 @@ organization-wide census. `docs/doctoring/actions-queue-saturation-hourly-sweep. `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` are *historical* RCA and plan/observation records for the `org-queue-sweep` that `#1878` removed; do not cite either, or these samples, as proof of current organization-wide occupancy. **There is no org-wide occupancy collector -in the tree today** — a narrow search of protected `main` finds no script in `scripts/ci/` reading -`runner_id`/`runner_name`, and `scripts/ci/audit_org_codeql_coverage.py` is a CodeQL *coverage* -census, not an occupancy collector. Organization-wide occupancy is therefore an open collection gap, +in the tree today** — no script under `scripts/` *reads* `runner_id` or `runner_name` +(`git grep -ln 'runner_id\|runner_name' origin/main -- scripts/` returns nothing), and +`scripts/ci/audit_org_codeql_coverage.py` is a CodeQL *coverage* census, not an occupancy collector. +Run that grep against `origin/main`, not a working branch: the same search on this branch matches +`audit_org_codeql_coverage.py:48`, which is the evidence comment added alongside this entry rather +than code reading either field. A search that your own edit satisfies has stopped being evidence. Organization-wide occupancy is therefore an open collection gap, recorded here as one rather than papered over with a sample. Nothing in this section closes a queue or 60-job definition of done. From 43ad43ce34e8869268f4111930723149ca2d9da4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 06:31:03 +0000 Subject: [PATCH 49/50] docs(triage): the governed path completed end to end at 06:23Z Run 34072122722 is the first opencode-review-dispatch run to finish after the allowlist reopened. Created 01:09:59Z for .github#1653 at head 2ee4c15b, it logged "Authorized repository_dispatch actor=opencode-agent[bot]", re-entered the queue at every needs: stage (01:58, 03:22, 04:42, 06:13), ran the review for 26 steps in 9m31s and published "Current-head formal OpenCode receipt id=5128650861 state=CHANGES_REQUESTED". End to end 5h13m, nearly all queue. That settles a question this catalog had left open: "the dispatch workflow will rerun this failed job" is a real recovery rather than a theoretical one, which is what makes the section 7 pacing rule matter -- a push inside that window restarts the chain. Section 1's expectation also held: the first post-fix verdict is CHANGES_REQUESTED with the deterministic fallback explicitly declining to approve over a Strix sandbox root cause, not an approval. Also recorded: a trap I fell into while establishing this. I used run_started_at != created_at as an execution test and got "0 of 43 ever started" -- for runs that had demonstrably executed. All 43 post-gate runs report run_started_at == created_at, the failures and the success alike. That discriminator is job-level only; at run level it is an artifact. The entry says to use steps > 0 on the jobs instead. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 426fb3cb3b..575cf3911e 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -222,6 +222,28 @@ it. What can be said without the hypothesis is narrower and solid: post-fix, a d longer evidence about the actor gate, and the throughput question is now queue depth, not authorization. +**Settled at 06:23Z: the path works end to end, and the earlier prediction held.** Run +`34072122722` is the first `opencode-review-dispatch` run to complete after the reopening. It was +created 01:09:59Z for `.github#1653` at head `2ee4c15b`, logged `Authorized repository_dispatch +actor=opencode-agent[bot]`, and each `needs:` stage re-entered the queue on its own — +`validate-pr-metadata` 01:58, `coverage-source-tree` 03:22, `coverage-evidence` 04:42, +`opencode-review` 06:13 — with the review itself running 26 steps in 9 m 31 s and publishing +`Current-head formal OpenCode receipt id=5128650861 state=CHANGES_REQUESTED`. **End to end: 5 h 13 m**, +almost all of it queue. So "the dispatch workflow will rerun this failed job" is a real recovery, not +a theoretical one — and the pacing rule in §7 is what makes it reachable, since any push in that +window restarts the whole chain. Note also that §1's expectation was borne out: the verdict is +`CHANGES_REQUESTED`, and the log shows the deterministic fallback explicitly declining to approve +(`MODEL_OUTPUT_UNAVAILABLE: deterministic evidence will not approve …`) over a Strix sandbox +root cause. First post-fix verdicts are refusals on heads carrying a stale red check, exactly as +predicted. + +**A measurement trap I fell into while establishing that, recorded so the next reader does not.** +`run_started_at != created_at` is a *job*-level discriminator and is meaningless at run level: of +those 43 post-gate runs, **all 43** report `run_started_at == created_at`, including the 10 that +failed and the 1 that succeeded — runs that demonstrably executed. Reading that as "0 of 43 ever +started" is an artifact, not a finding. Use `steps > 0` on the run's *jobs*, as the occupancy +paragraph in §7 says, and never the run-level timestamps. + **Do not.** Do not reflexively re-run the failed job by hand, and do not "fix" the PR's code — this failure says nothing about it. From 727bb6477fc0b038fe2ec6b49d3f7925c9af28a6 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 7 Sep 2026 07:10:58 +0000 Subject: [PATCH 50/50] docs(triage): observe the composite sandbox verdict in a live artifact The correction earlier in this batch -- that the sandbox verdict is one composite line beginning with the provider token, so a prefix test misfiles every sandbox run -- was derived from strix_quick_gate.sh:4393 rather than from an artifact. .github#1482's strix job 101632714331 (head 2e92e82f) now shows it, verbatim: Retrying model 'orchestrator/free' due to Caido sandbox bootstrap timing (attempt 2/2). STRIX_PROVIDER_UNAVAILABLE: STRIX_SANDBOX_UNAVAILABLE: the last Strix attempt ended in the sandbox bootstrap ... So the rule now rests on source and observation together, and the entry shows a live run that a prefix test would file as gateway exhaustion. The retry budget also reads attempt 2/2 against the pre-#1953 specimen's 3/3, which is #1953 giving the sandbox class its own bounded retry instead of borrowing the transient-error budget. No action taken on #1482 -- an owner-hand-driven Draft carrying the "Current exact authority" marker. Gate: 2975 passed, 1 skipped, 21 subtests, coverage 100%, interrogate 100%. Refs #1953. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- docs/doctoring/ci-failure-signature-triage.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/doctoring/ci-failure-signature-triage.md b/docs/doctoring/ci-failure-signature-triage.md index 575cf3911e..e6cec8eb79 100644 --- a/docs/doctoring/ci-failure-signature-triage.md +++ b/docs/doctoring/ci-failure-signature-triage.md @@ -660,7 +660,23 @@ kept deliberately so the workflow still classifies a finding-free sandbox outage infrastructure evidence. A test of the form "the line starts with `STRIX_PROVIDER_UNAVAILABLE` → gateway" therefore misfiles **every** sandbox run. Search for `STRIX_SANDBOX_UNAVAILABLE` as a substring anywhere in the line, which is what the one consumer that reads it correctly does -(`opencode-review-dispatch.yml:5648`). A fourth specimen closed the set the same day: `#1930`'s run +(`opencode-review-dispatch.yml:5648`). + +**Observed, not only derived from the source.** `.github#1482`'s `strix` job `101632714331` +(head `2e92e82f`, 2026-09-07) ends exactly as `strix_quick_gate.sh:4393` predicts: + +``` +Retrying model 'orchestrator/free' due to Caido sandbox bootstrap timing (attempt 2/2). +… +STRIX_PROVIDER_UNAVAILABLE: STRIX_SANDBOX_UNAVAILABLE: the last Strix attempt ended in the sandbox +bootstrap (Caido proxy on 127.0.0.1 unreachable through Strix's loginAsGuest attempts) after 1 +sandbox-specific same-model retries +``` + +A prefix test on that line files this run as gateway exhaustion; a substring test files it correctly +as the sandbox. Note the retry budget reads `attempt 2/2` here against the pre-`#1953` specimen's +`3/3`, which is `#1953` giving the sandbox class its own bounded retry rather than borrowing the +transient-error budget. A fourth specimen closed the set the same day: `#1930`'s run `34027404208` carries `loginAsGuest` in `strix-pr-scope-jibt1j_66a8/strix.log` yet ends `STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted`, with a healthy `ready 6 / probed 16 / escalations_used 2` preflight — a grep files it as a sandbox outage, the