Skip to content

chore(ci): move :prod promotion to Mon-Thu 12:00 PT - #875

Open
Evrard-Nil wants to merge 2 commits into
mainfrom
chore/prod-promote-mon-thu-noon-pt
Open

chore(ci): move :prod promotion to Mon-Thu 12:00 PT#875
Evrard-Nil wants to merge 2 commits into
mainfrom
chore/prod-promote-mon-thu-noon-pt

Conversation

@Evrard-Nil

Copy link
Copy Markdown
Contributor

What

Move the automated :staging -> :prod promotion off its weekly Thursday 00:00 UTC cron to Mon-Thu 19:00 UTC (12:00 PT during PDT).

cron: "0 0 * * 4" -> cron: "0 19 * * 1-4"

Why

Brave verifies each NEAR release and denies traffic to us when verification fails (verifiable privacy is their core feature). A prod promotion landing overnight UTC gives them no US-hours window to run the code audit and verifier. On 2026-07-09 an unverified build promoted overnight and Brave cut the connection until we rolled back.

Per Lloyd's proposed schedule in the Brave thread: Mon-Thu, staging freeze at 10:00 PT, prod promotion at 12:00 PT, so every release gets a predictable ~2h verification slot during US business hours.

This is the first of two coordinated changes:

  • this PR - cloud-api promote.yml retags :staging -> :prod at 12:00 PT.
  • cvm-ansible-playbooks update_cloud_api_prod.yml - rolls the promoted image onto cpu01,cpu02 shortly after (companion PR linked below).

Notes / caveats

  • DST: GitHub cron is UTC-only. 19:00 UTC = 12:00 PT in summer (PDT) and 11:00 PT in winter (PST). Revisit the offset at the November DST change (switch to 20:00 UTC to hold 12:00 PT).
  • Daily Mon-Thu cadence also needs the build.yml staging-freeze logic updated: today it freezes :staging on Tue/Wed (the old once-weekly model), which would suppress new staging builds on those days. Reworking that freeze (time-of-day 10:00 PT window vs. day-of-week) needs Lloyd's sign-off on exact semantics and is a follow-up, not in this PR.
  • Manual workflow_dispatch promotion is unchanged.
  • Not urgent to merge until the release schedule is confirmed with Brave.

Promote :staging to :prod at 19:00 UTC Mon-Thu (12:00 PT / PDT) instead
of the weekly Thursday 00:00 UTC cron, so each release gets a US-hours
window for Brave's code audit and verifier before it goes to prod.
@cursor

cursor Bot commented Jul 9, 2026

Copy link
Copy Markdown

Bugbot is not enabled for this team, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Moves the scheduled GitHub Actions promotion workflow so the :staging:prod retag happens during a predictable US-hours window, aligning with Brave’s verification/audit process.

Changes:

  • Update promote.yml cron from weekly Thu 00:00 UTC to Mon–Thu 19:00 UTC.
  • Expand in-file scheduling documentation (including UTC-only cron/DST caveat).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

schedule:
# Thursday 00:00 UTC — promote frozen :staging → :prod
- cron: "0 0 * * 4"
# Mon-Thu 19:00 UTC (12:00 PT during PDT): promote verified :staging to :prod.
@claude

claude Bot commented Jul 9, 2026

Copy link
Copy Markdown

Review — chore(ci): move :prod promotion to Mon-Thu 12:00 PT

The cron change itself is correct: 0 19 * * 1-4 = 19:00 UTC Mon–Thu, and the DST note (19:00 UTC → 12:00 PT PDT / 11:00 PST) is accurate. Comments are clear.

⚠️ One critical sequencing concern (production safety)

promote.yml has no verification gate — on schedule it unconditionally copies whatever is on :staging:prod. The only thing that makes the promoted image "verified/frozen" is build.yml's freeze, which skips :staging only on Tue(2)/Wed(3) (build.yml:65). Against the new Mon–Thu promotion, the days no longer line up:

  • Mon & Thu — staging is not frozen, so :staging carries the latest fresh main build. promote.yml will push an unverified build to :prod — i.e. exactly the overnight-unverified-promotion failure this PR sets out to prevent (just at noon instead of midnight).
  • Tue & Wed — staging frozen, so both days re-promote the same digest (extra stamped tags + GH releases for identical bytes; harmless but noisy).

You already flag the build.yml coupling in the caveats and say "not urgent to merge until the release schedule is confirmed with Brave" — so this isn't news. The point is just to make the merge ordering explicit: landing this PR before the build.yml freeze rework leaves prod actively promoting unverified builds Mon & Thu, arguably a regression on the current once-weekly-Thu behavior. Recommend gating this merge on (or co-landing with) the freeze-window change so the freeze days cover the full Mon–Thu promotion window.

No blocking issues in the diff as written. ⚠️ (hold merge until the build.yml freeze logic is aligned)

@PierreLeGuen PierreLeGuen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schedule-only change to .github/workflows/promote.yml: the :prod promotion cron moves to 0 19 * * 1-4 (Mon–Thu 19:00 UTC / 12:00 PT). The cron is well-formed and the rest of the workflow is untouched.

No blocking issues. Non-blocking notes:

  • Firing 4×/week instead of weekly is safe: the prod-image-mutation concurrency group (cancel-in-progress: false) serializes runs, and re-promoting a frozen :staging digest is idempotent (skopeo copy overwrites with identical bytes, tag/release steps re-derive the same values).
  • The DST caveat (fixed UTC offset) is documented in-file and in the PR body — a known operational note, not a bug.
  • The build.yml staging-freeze interaction on Tue/Wed is explicitly deferred to a follow-up and does not corrupt state.

Checks: confirmed the diff touches only promote.yml; validated cron 0 19 * * 1-4; reasoned through promotion idempotency and concurrency under the new cadence; reviewed prior comments (Copilot informational only). git diff --check passed; actionlint not installed (skipped).

@lloydmak99 lloydmak99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schedule-only change to .github/workflows/promote.yml: cron: "0 0 * * 4"cron: "0 19 * * 1-4" plus expanded comments. The cron is well-formed and correctly fires Mon–Thu at 19:00 UTC (12:00 PDT summer / 11:00 PST winter, matching the stated 12:00 PT window). Rest of the workflow is untouched and the prod-image-mutation concurrency group (cancel-in-progress: false) still serializes the more frequent runs.

  • Optional follow-up, .github/workflows/promote.yml:12 — with the new Mon–Thu cadence, promotion is still unconditional while build.yml:65 only freezes :staging Tue/Wed, so Mon & Thu could promote fresh, unverified builds. Pre-existing behavior, already noted in the PR body as a deferred freeze-window rework; not a blocker for this diff.

Checks: git diff --check clean (no whitespace issues); promote.yml parses via yaml.safe_load; cron syntax and DST offsets validated manually. actionlint not run (unavailable in-environment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants