Skip to content

fix(watchdog): don't treat a mid-run mirror as an emergency #2578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Hugo0 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(watchdog): don't treat a mid-run mirror as an emergency #2578

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions .github/workflows/content-pipeline-watchdog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,13 +116,20 @@ jobs:
fi

STALE=""
MIRROR_BROKEN=false
# URGENT = skip the grace period. Reserve it for states that cannot
# resolve on their own. A mirror that is merely mid-run is NOT one of
# them: it finishes in ~30s, and treating it as urgent turned every
# push landing shortly before a tick into a false alarm. That fired
# for real (16:04:13 run, 21s after a push, 12s before the mirror
# completed — "Stuck for 5m", and green again on the next tick).
# A genuinely dead mirror still alerts; it just waits out the grace,
# which is exactly what the grace period is for.
URGENT=false
if [ "$DEGRADED" = true ]; then
STALE="**the watchdog itself could not evaluate the pipeline** — check its permissions before assuming content is stuck"
MIRROR_BROKEN=true
URGENT=true
elif [ "${EXPECTED:0:7}" != "${MIRRORED:0:7}" ]; then
STALE="the mirror is behind mono (expected \`${EXPECTED:0:7}\`, published \`${MIRRORED:0:7}\`)"
MIRROR_BROKEN=true
elif [ "$LIVE" != "$TIP" ]; then
STALE="production is behind peanut-content (live \`${LIVE:0:7}\`, latest \`${TIP:0:7}\`)"
fi
Expand All @@ -139,8 +146,7 @@ jobs:
--json number,headRefName \
--jq '[.[] | select((.headRefName | startswith("auto/update-content-")) or (.headRefName | startswith("content/publish-to-main")))] | first | .number // empty' || true)

# A failed mirror will not fix itself, so skip the grace period.
DOOMED=$MIRROR_BROKEN
DOOMED=$URGENT

@coderabbitai coderabbitai Bot Jul 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Anchor mirror grace to the newly detected mono change.

DOOMED=$URGENT removes the explicit bypass, but the grace check still uses AGE from TIP_AT. If the mirror’s previous publish is already older than 45 minutes, a fresh mono change makes the watchdog alert immediately on the first tick, so the false alarm remains possible. Track the selected EXPECTED commit timestamp and use it for mirror-behind age; keep TIP_AT for production-behind age. The updated comment is otherwise misleading because this state does not always receive the full window.

Also applies to: 160-163

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/content-pipeline-watchdog.yml at line 149, Update the
watchdog’s mirror-behind grace calculation around DOOMED, EXPECTED, and TIP_AT
to track the timestamp of the selected EXPECTED commit and use it when computing
mirror-behind AGE; continue using TIP_AT for production-behind age. Preserve the
full grace window for a newly detected mono change and revise the nearby comment
to match the resulting behavior.

@chip-peanut-bot chip-peanut-bot Bot Aug 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAJOR: Clock mirror grace from the triggering mono commit

When the mirror has been idle longer than 45 minutes and a new mono content commit lands just before this job, EXPECTED changes but TIP_AT still describes the previous mirror commit. This line makes the mismatch non-urgent, yet AGE remains over 45 because it is clocked from TIP_AT, so the grace guard is skipped and the watchdog posts the same false alarm while the new mirror run is in flight. Fetch the expected mono commit timestamp and use that age for mirror-behind (keeping TIP_AT for production-behind), or otherwise maintain per-hop clocks.

if [ -n "$NUM" ]; then
PR_SHA=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/pulls/$NUM" --jq '.head.sha' || true)
CI=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/commits/$PR_SHA/check-runs?per_page=100" \
Expand All @@ -151,7 +157,10 @@ jobs:
WHERE="no publish PR is open — the mirror, the dispatch, or update-content.yml failed"
fi

# Red CI never self-resolves, so don't wait out the grace period.
# Only genuinely unrecoverable states skip the grace period: red CI,
# and a watchdog that cannot evaluate itself. Everything else gets
# the full window, because most staleness at this point is just the
# pipeline mid-flight.
if [ "$DOOMED" = false ] && [ "$AGE" -lt "$GRACE_MINUTES" ]; then
echo "$STALE — ${AGE}m old, within ${GRACE_MINUTES}m grace ($WHERE). Not alerting yet."
exit 0
Expand Down
Loading