fix(watchdog): don't treat a mid-run mirror as an emergency - #2578
Conversation
The watchdog alerted 21 seconds after a content push, 12 seconds before the
mirror finished, and was green again on the next tick:
16:03:52 push f3008839
16:04:13 watchdog: "the mirror is behind mono ... Stuck for 5m"
16:04:25 mirror completes
16:43:38 green
A mirror-behind state skipped the 45-minute grace period, on the reasoning that
"a failed mirror will not fix itself". That reasoning was wrong. A mirror that is
merely mid-run fixes itself in about 30 seconds, and the two states look
identical from outside. So any push landing shortly before a */15 tick produced
a false alarm — roughly one in twenty pushes, indefinitely.
Skipping the grace period is now reserved for states that genuinely cannot
recover: red CI, and a watchdog that cannot evaluate itself. A dead mirror still
alerts; it waits out the window first, which is what the window is for.
Verified the full decision table:
mirror mid-flight, 5m -> SILENT (was: ALERT — the bug)
mirror stuck, 60m -> ALERT
prod behind, 5m -> SILENT
prod behind, 60m -> ALERT
prod behind, CI RED -> ALERT immediately
watchdog degraded -> ALERT immediately
healthy -> SILENT
Also ran the real step under `bash -e` against live APIs: reports current, exits 0.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThe watchdog workflow replaces ChangesWatchdog grace-period classification
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Code-analysis diffPainscore total: 6308.27 → 6308.27 (0) |
🧪 UI test report — ✅ all greenSuites
📊 Coverage (unit)
⏱ 10 slowest test cases
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/content-pipeline-watchdog.yml:
- Line 149: Update the watchdog’s mirror-behind grace calculation around DOOMED,
EXPECTED, and TIP_AT to track the timestamp of the selected EXPECTED commit and
use it when computing mirror-behind AGE; continue using TIP_AT for
production-behind age. Preserve the full grace window for a newly detected mono
change and revise the nearby comment to match the resulting behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 67c5ecea-8ce3-4fac-9f74-e2d0a71cf738
📒 Files selected for processing (1)
.github/workflows/content-pipeline-watchdog.yml
|
|
||
| # A failed mirror will not fix itself, so skip the grace period. | ||
| DOOMED=$MIRROR_BROKEN | ||
| DOOMED=$URGENT |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Anchor mirror grace to the newly detected mono change.
DOOMED=$URGENT removes the explicit bypass, but the grace check still uses AGE from TIP_AT. If the mirror’s previous publish is already older than 45 minutes, a fresh mono change makes the watchdog alert immediately on the first tick, so the false alarm remains possible. Track the selected EXPECTED commit timestamp and use it for mirror-behind age; keep TIP_AT for production-behind age. The updated comment is otherwise misleading because this state does not always receive the full window.
Also applies to: 160-163
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/content-pipeline-watchdog.yml at line 149, Update the
watchdog’s mirror-behind grace calculation around DOOMED, EXPECTED, and TIP_AT
to track the timestamp of the selected EXPECTED commit and use it when computing
mirror-behind AGE; continue using TIP_AT for production-behind age. Preserve the
full grace window for a newly detected mono change and revise the nearby comment
to match the resulting behavior.
![chip-peanut-bot[bot]](https://avatars.githubusercontent.com/in/2915219?s=60&v=4){kind=small}
There was a problem hiding this comment.
Chip review — changes requested
Request changes: the mirror-behind path now enters the grace gate, but the gate still uses the age of the previous mirror commit, so fresh pushes after an idle period still false-alert.
Findings
- MAJOR · .github/workflows/content-pipeline-watchdog.yml:149 · Clock mirror grace from the triggering mono commit
When the mirror has been idle longer than 45 minutes and a new mono content commit lands just before this job, EXPECTED changes but TIP_AT still describes the previous mirror commit. This line makes the mismatch non-urgent, yet AGE remains over 45 because it is clocked from TIP_AT, so the grace guard is skipped and the watchdog posts the same false alarm while the new mirror run is in flight. Fetch the expected mono commit timestamp and use that age for mirror-behind (keeping TIP_AT for production-behind), or otherwise maintain per-hop clocks.
Checked clean
- Verified the detached worktree is exactly the supplied head and the supplied base is its merge base.
- Reviewed healthy, degraded, mirror-behind, production-behind, red-CI, no-publish-PR, and grace-boundary state transitions.
- Confirmed from the mono mirror workflow that a mono push precedes creation of the new peanut-content commit, so TIP_AT can be arbitrarily old during a fresh mirror mismatch.
- Checked exact-head CI: typecheck, unit, e2e, format, CodeQL, and aggregate gates passed; direct eslint failures also occur on the base SHA and are unrelated to this YAML-only change.
- Checked permissions, trigger trust, secret handling, command injection, duplication, naming, and architecture drift; no additional actionable defect survived.
Second opinion by moonshotai/kimi-k3: 0 finding(s), marked with the model name. It reads the diff only, so treat its findings as advice.
Exact head: f7241bd31519 · Context: repo, mono
|
|
||
| # A failed mirror will not fix itself, so skip the grace period. | ||
| DOOMED=$MIRROR_BROKEN | ||
| DOOMED=$URGENT |
There was a problem hiding this comment.
MAJOR: Clock mirror grace from the triggering mono commit
When the mirror has been idle longer than 45 minutes and a new mono content commit lands just before this job, EXPECTED changes but TIP_AT still describes the previous mirror commit. This line makes the mismatch non-urgent, yet AGE remains over 45 because it is clocked from TIP_AT, so the grace guard is skipped and the watchdog posts the same false alarm while the new mirror run is in flight. Fetch the expected mono commit timestamp and use that age for mirror-behind (keeping TIP_AT for production-behind), or otherwise maintain per-hop clocks.
1 participant
The watchdog false-alarmed in Discord today. Content was fine — it caught the pipeline mid-flight:
Cause
Mirror-behind skipped the 45-minute grace period, on my reasoning that "a failed mirror will not fix itself." That reasoning was wrong. A mirror that is merely mid-run fixes itself in ~30 seconds, and from outside it is indistinguishable from a dead one.
So any content push landing shortly before a
*/15tick produces a false alarm — roughly 1 in 20 pushes, indefinitely. That is exactly the alert-fatigue pattern that makes a watchdog worthless.Fix
Skipping the grace period is now reserved for states that genuinely cannot recover: red CI, and a watchdog that cannot evaluate itself. A dead mirror still alerts — it just waits out the window first, which is what the window is for.
Decision table, verified
Also ran the real step under
bash -eagainst live APIs — reportsproduction content is current, exit 0.Note
This is a false-alarm fix, not a pipeline fix. Content publishing itself is working: three consecutive autonomous publishes today (#2561, #2575, #2576), all merged by the automation, production in sync.
Summary by CodeRabbit