Skip to content

ci: run independent workflow steps in parallel #518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
caugner wants to merge 5 commits into
base: main
Choose a base branch
from
+303 −294
Open

ci: run independent workflow steps in parallel #518

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bf3f0a0
ci(prod-build): parallelize checkouts + index updates
caugner Sep 10, 2026
d4e4cf7
ci(stage-build): parallelize checkouts + index updates
caugner Sep 10, 2026
5d4fa66
ci(test-build): parallelize checkouts
caugner Sep 10, 2026
e1a8a04
ci(prod-build,stage-build): guard AI Help index step with `SKIP_BUILD`
caugner Sep 10, 2026
251b573
ci(prod-build,stage-build): make AI Help index step `continue-on-error`
caugner Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
206 changes: 105 additions & 101 deletions .github/workflows/prod-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,68 +46,85 @@ jobs:
if: github.repository == 'mdn/dex'

steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: mdn/dex
persist-credentials: false

- name: Checkout (content)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
with:
repository: mdn/content
path: mdn/content
# Yes, this means fetch EVERY COMMIT EVER.
# It's probably not sustainable in the far future (e.g. past 2021)
# but for now it's good enough. We'll need all the history
# so we can figure out each document's last-modified date.
fetch-depth: 0
persist-credentials: false

- name: Checkout (blog)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/blog
path: mdn/blog
lfs: true
token: ${{ secrets.MDN_STUDIO_PAT }}
persist-credentials: false

- name: Checkout (generic-content)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/generic-content
path: mdn/generic-content
persist-credentials: false

- name: Checkout (curriculum)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/curriculum
path: mdn/curriculum
persist-credentials: false

- name: Checkout (translated-content)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
with:
repository: mdn/translated-content
path: mdn/translated-content
# See matching warning for mdn/content checkout step
fetch-depth: 0
persist-credentials: false

- name: Checkout (translated-content-de)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
with:
repository: mdn/translated-content-de
path: mdn/translated-content-de
persist-credentials: false
- parallel:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: mdn/dex
persist-credentials: false

- name: Checkout (content)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
with:
repository: mdn/content
path: mdn/content
# Yes, this means fetch EVERY COMMIT EVER.
# It's probably not sustainable in the far future (e.g. past 2021)
# but for now it's good enough. We'll need all the history
# so we can figure out each document's last-modified date.
fetch-depth: 0
persist-credentials: false

- name: Checkout (blog)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/blog
path: mdn/blog
lfs: true
token: ${{ secrets.MDN_STUDIO_PAT }}
persist-credentials: false

- name: Checkout (generic-content)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/generic-content
path: mdn/generic-content
persist-credentials: false

- name: Checkout (curriculum)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/curriculum
path: mdn/curriculum
persist-credentials: false

- name: Checkout (translated-content)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
with:
repository: mdn/translated-content
path: mdn/translated-content
# See matching warning for mdn/content checkout step
fetch-depth: 0
persist-credentials: false

- name: Checkout (translated-content-de)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
with:
repository: mdn/translated-content-de
path: mdn/translated-content-de
persist-credentials: false

- name: Checkout (mdn-contributor-spotlight)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/mdn-contributor-spotlight
path: mdn/mdn-contributor-spotlight
persist-credentials: false

- name: Checkout (fred)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/fred
path: mdn/fred
persist-credentials: false

- name: Move de into translated-content
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
Expand All @@ -122,22 +139,6 @@ jobs:
git add files/de
git -c user.name='MDN' -c user.email='mdn-dev@mozilla.com' commit -m 'de'

- name: Checkout (mdn-contributor-spotlight)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/mdn-contributor-spotlight
path: mdn/mdn-contributor-spotlight
persist-credentials: false

- name: Checkout (fred)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: ${{ ! vars.SKIP_BUILD }}
with:
repository: mdn/fred
path: mdn/fred
persist-credentials: false

- name: Setup Node (fred)
if: ${{ ! vars.SKIP_BUILD || ! vars.SKIP_FUNCTION }}
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
Expand Down Expand Up @@ -321,14 +322,6 @@ jobs:
# Use a GITHUB_TOKEN to bypass rate limiting for rari.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Update search index
if: ${{ ! vars.SKIP_BUILD }}
working-directory: mdn/dex/deployer-js
env:
DEPLOYER_ELASTICSEARCH_URL: ${{ secrets.DEPLOYER_PROD_ELASTICSEARCH_URL }}
FRED_ROOT: ${{ github.workspace }}/mdn/fred
run: node main.js search-index "$FRED_ROOT/out"

- name: Authenticate with GCP
if: ${{ ! vars.SKIP_BUILD }}
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0
Expand All @@ -341,13 +334,32 @@ jobs:
if: ${{ ! vars.SKIP_BUILD }}
uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db # v3.0.1

- name: Sync build
if: ${{ ! vars.SKIP_BUILD }}
working-directory: mdn/fred
run: |-
time gsutil -q -m -h "Cache-Control: public, max-age=3600" cp -r out/static gs://${{ vars.GCP_BUCKET_NAME }}/main/
time gsutil -q -m -h "Cache-Control: public, max-age=3600" rsync -cr gs://${{ vars.GCP_BUCKET_NAME }}/main/static/ gs://${{ vars.GCP_BUCKET_NAME }}/fred/static/
time gsutil -q -m -h "Cache-Control: public, max-age=3600" rsync -cdrj html,json,txt -y "^static/" out gs://${{ vars.GCP_BUCKET_NAME }}/fred
- parallel:
- name: Update search index
if: ${{ ! vars.SKIP_BUILD }}
working-directory: mdn/dex/deployer-js
env:
DEPLOYER_ELASTICSEARCH_URL: ${{ secrets.DEPLOYER_PROD_ELASTICSEARCH_URL }}
FRED_ROOT: ${{ github.workspace }}/mdn/fred
run: node main.js search-index "$FRED_ROOT/out"

- name: Update AI Help index with macros
if: ${{ ! vars.SKIP_BUILD }}
continue-on-error: true

@LeoMcA LeoMcA Sep 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm not so sure about this. Before, we had a nice situation where a failed ai help index would fail the build (alerting us), but not fail to update the content.

Now, do we either have to chose between us finding out, or us bringing down the content upload.

I think I'd move this back, it didn't really count in the "time from start to deploy" sense

working-directory: mdn/dex
run: npm run ai-help-macros -- update-index "$FRED_ROOT/out/en-us/docs"
env:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
PG_URI: ${{ secrets.PG_URI }}
FRED_ROOT: ${{ github.workspace }}/mdn/fred

- name: Sync build
if: ${{ ! vars.SKIP_BUILD }}
working-directory: mdn/fred
run: |-
time gsutil -q -m -h "Cache-Control: public, max-age=3600" cp -r out/static gs://${{ vars.GCP_BUCKET_NAME }}/main/
time gsutil -q -m -h "Cache-Control: public, max-age=3600" rsync -cr gs://${{ vars.GCP_BUCKET_NAME }}/main/static/ gs://${{ vars.GCP_BUCKET_NAME }}/fred/static/
time gsutil -q -m -h "Cache-Control: public, max-age=3600" rsync -cdrj html,json,txt -y "^static/" out gs://${{ vars.GCP_BUCKET_NAME }}/fred

- name: Authenticate with GCP
if: ${{ ! vars.SKIP_FUNCTION }}
Expand Down Expand Up @@ -418,14 +430,6 @@ jobs:
wait $pid
done

- name: Update AI Help index with macros
working-directory: mdn/dex
run: npm run ai-help-macros -- update-index "$FRED_ROOT/out/en-us/docs"
env:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
PG_URI: ${{ secrets.PG_URI }}
FRED_ROOT: ${{ github.workspace }}/mdn/fred

- name: Slack Notification
if: failure()
uses: rtCamp/action-slack-notify@33ca3be66c6f378fe1610fd1d5258632dbed5e58 # v2
Expand Down
Loading
Loading