feat(ci): rerun a red PR's failed Actions jobs once before a fix round (#487)

[mabry1985]

Fixes #487

What

A red PR used to spend a coder fix round (and a ci_fix_max unit) right away, even when the red was a flaky job. Now the CI reconcile reruns the failed GitHub Actions jobs first, once per PR head, and bounces only if they fail again.

Design

• Seam: worktree.rerun_failed_ci(pr_url="", *, cwd, run_ids=None, slug=""). It reads the PR's statusCheckRollup and uses the pure failed_ci_run_ids(checks) to get (owner/repo, run_id) for each Actions run behind a failed blocking check. It parses the run from detailsUrl /actions/runs/<id> and dedupes per run. It then runs gh run rerun <id> --failed -R <slug> for each and returns the ids GitHub accepted. Passing run_ids= skips the rollup read, which is how the real tier points it at the pinned fixture run. It is best-effort: a gh error, timeout or refusal is logged, that run is left out, and the function never raises. [] means "bounce as before".
• Loop: _reconcile_ci. On failing, _rerun_ci_once runs first:
  • It reads the card's ci-rerun:<sha12>:<n> stamp. If there is a stamp, it reads pr_head_sha. A different head gets a fresh allowance. The same head with n >= ci_rerun_max bounces as today. An unreadable head also bounces as today.
  • Otherwise it reruns. If run ids come back, it stamps the head, logs CI red at <head> — rerunning failed jobs once before a fix round: <run ids>, and returns. There is no requeue, no ci-fix spend and no feedback injection.
  • With no stamp, the head is read after a successful rerun, so a red card with no Actions runs costs no extra gh call.
• On passing with a stamp, _settle_ci_rerun logs one CI flake line naming the checks that failed, if the head is still the one that was rerun. It then clears the stamp. A green new head clears it silently.
• Stamp: store.record_ci_rerun(fid, head="", n=1). This is the single-label-replaced pattern via replace_prefixed_label_args, plus ci_rerun_from_labels. The label is at most 23 chars, well under beads' 50-char cap. It lives on the bead, so a restart never reruns a head twice. The format is :<n> rather than a bare sha so ci_rerun_max > 1 works.
• Config: ci_rerun_max. Default 1; 0 turns it off. It is read like ci_fix_max in loop/core.py and documented in docs/configuration.md (row + paragraph; the YAML-only count is now 29 of 80). It is also in the manifest config: block beside ci_fix_max and in the Settings fields. The README has no ci_fix_max entry, so the manifest's commented config is the matching place. Rerunning needs actions: write on the board's gh token.

Fixture and real tier

• The external-seam ratchet: rerun_failed_ci is REAL, so the worktree contract goes from 42 to 43 REAL. The new store seam record_ci_rerun is REAL too; the store UNCOVERED count stays 16.
• tests/test_publish_gate_real.py::test_rerun_failed_ci_reruns_the_pinned_failed_fixture_run calls worktree.rerun_failed_ci(run_ids=[PB_GH_FIXTURE_FAILED_RUN], slug=...) on the pinned ci-fixture-fails.yml run.
  • It first waits, for up to 180s, until the run is completed, so another CI run's rerun can't collide. If a collision still wins, the seam returns [], and the test accepts that as long as run_attempt still rose.
  • It then asserts run_attempt increases, polling for up to 60s. It does not wait for the new attempt to finish.
  • It is gated on PB_GH_ALLOW_WRITES. With writes allowed under PB_REQUIRE_GH, a missing run id fails rather than skips.
• A second real test checks that a bogus run id (a real 404) returns [] without raising.
• ci.yml's test (real gh) job now has actions: write and PB_GH_FIXTURE_FAILED_RUN: ${{ vars.PB_GH_FIXTURE_FAILED_RUN }}. The ratchet asserts both.
• conftest.py: an autouse stub makes rerun_failed_ci return [] in the unit tier, so every existing CI-bounce test bounces unchanged. The real seam goes into REAL_SEAMS and is restored by the tier's real_seams fixture.

Unit tests (tests/test_ci_rerun_487.py)

These use a mocked gh in the _stub_ci_worktree style:

• the first red reruns, with no requeue and no budget spent;
• green after a rerun logs one flake line naming the checks and clears the stamp;
• red again at the same head bounces;
• the allowance survives a restart;
• a new head gets a new allowance;
• no Actions run ids bounces immediately;
• ci_rerun_max: 0 is the old behavior; ci_rerun_max: 2 reruns a head twice;
• a green new head clears the stamp without logging a flake;
• the rollup parse (skips advisory App checks, non-Actions statuses and green runs);
• the seam's per-run refusal and exception handling;
• the ci-rerun: stamp through real br: it stamps, re-stamps idempotently, is replaced rather than accumulated, fits the 50-char cap, and clears.

Verification

• Full suite: 2692 passed, 40 skipped. ruff check . is clean and ruff format --check passes on the changed files.
• Real tier run locally with PB_REQUIRE_GH=1 PB_GH_FIXTURE_PR=…/pull/370 PB_GH_FIXTURE_FAILED_RUN=36520566451 PB_GH_ALLOW_WRITES=1: both rerun tests passed. Fixture run 36520566451 went from attempt 1 to attempt 2.

🤖 Generated with Claude Code

[protoreview]

QA panel review — PASS

code-review-structural · head bd28dcc3351e · formal

No findings were raised, so there was nothing to verify. The panel (correctness, removed-behavior, cross-file, conventions, and protoPatch structural) reviewed the CI-rerun state machine, allowance tracking, cross-file references, and hygiene conventions across all changed files and found no defects. The one protoPatch nit (falsy timeout=0 in dispatch_coder) targets pre-existing code outside this PR's scope and was dropped. No coverage gaps noted.

No findings — the review came back clean.

findings JSON (machine-readable)

[]

[protoreview]

Promoting the PASS verdict for head bd28dcc3351e: all checks terminal-green, zero unresolved review threads. (approve-on-green)