-
Notifications
You must be signed in to change notification settings - Fork 6.1k
feat(ci): add fix-flakes workflow to auto-fix flaky tests #41692
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 0 additions & 6 deletions
6
.claude/skills/playwright-bot-voice/SKILL.md → .github/workflows/bot-voice.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| # Playwright: Fix a Flaky or Red Test | ||
|
|
||
| Turn CI test-results data into **one** concrete fix: pick a high-impact flaky-or-red test that is reproducible on your OS, | ||
| confirm nobody's on it, fix the root cause *or* scope a skip, pick a reviewer, and hand off a | ||
| single commit that becomes the PR. Fully autonomous — no approval stops. | ||
|
|
||
| ## 1. Pick one target | ||
|
|
||
| Query the DB following the patterns in `.claude/skills/playwright-test-results/SKILL.md`. | ||
| Two families: | ||
|
|
||
| - **Cross-run flake** — the *final* verdict (after retries) flips between runs. | ||
| - **Consistent red** — `expected_status = 'passed'` yet failing in ~every run. | ||
|
|
||
| Rank by **impact**: fail %, run count (floor `runs >= 10` so it isn't a one-off), and how many | ||
| bots/PRs it disrupts — **not** by "has a tidy error message". | ||
| Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it. | ||
| Keep the ranked list — you fall back down it in step 2. Say why you picked the top one. | ||
|
|
||
| ## 2. Check nobody's already on it | ||
|
|
||
| Search open PRs/issues for the test title words and file path before touching anything. | ||
| Also check any issue linked in the | ||
| test's `annotation`. | ||
| If an open PR already touches the test, **it's taken — drop it and go down your step-1 ranking**, re-checking each. | ||
| Only stop and report if the whole shortlist is covered. | ||
|
|
||
| ## 3. Reproduce on this OS | ||
|
|
||
| Read the test and its `error_message`. | ||
|
|
||
| Build first (`npm run build`; watch is **not** running — if you touch generated-code files see | ||
| `CLAUDE.md`). Then reproduce, scoped to the failing target. | ||
|
|
||
| The DB `project_name` **is** the Playwright `--project`; `bot_name` encodes the OS. **Always | ||
| pass a `<file>:<line>` filter** — a bare run launches the whole suite. The browser test scripts | ||
| are project-locked, so use the one matching the failing bot: | ||
|
|
||
| | Failing target | Command | | ||
| |---|---| | ||
| | chromium | `npm run ctest -- <file>:<line>` | | ||
| | firefox | `npm run ftest -- <file>:<line>` | | ||
| | webkit | `npm run wtest -- <file>:<line>` | | ||
| | `tests/playwright-test/**` | `npm run ttest -- <file>:<line>` | | ||
| | `tests/mcp/**` | `npm run test-mcp -- --project=<name> <file>:<line>` | | ||
|
|
||
| Other suites (electron `etest`, etc.): see `package.json` scripts. | ||
| Add `--repeat-each=N` to force a flake's flip. | ||
|
|
||
| **You can only reproduce what this OS reaches.** A failure you can't reach here is itself | ||
| evidence it's environment-specific — a skip candidate (step 6), not a blind-fix. | ||
| Keep any OS handling keyed on the *current* OS, never hardcoded. | ||
| Broader OS coverage comes from different agent runs on other OSes, not from you. | ||
|
|
||
| ## 4. Fix — root cause or scoped skip | ||
|
|
||
| - **Tractable → fix the source.** Usually a test-side race: a missing `await`, waiting on the | ||
| wrong signal, an under-specified locator, leaked state. Fix the test (or the product bug). | ||
| - **Engine/OS-specific, unreachable here, or genuinely hard → scope a skip**, narrowed to the | ||
| exact failing condition — never a whole file or browser: | ||
|
|
||
| ```ts | ||
| it.fixme(browserName === 'webkit' && isLinux, 'https://github.com/microsoft/playwright/issues/NNNNN'); | ||
| ``` | ||
|
|
||
| Link an issue if one exists or explain why the test is skipped. **Default to `fixme`** — it parks | ||
| the debt and stays greppable. Use `skip` only if reproduction shows the test is genuinely | ||
| mis-scoped for that config (`skip` claims "this failure is expected and correct," which a | ||
| flake-fighter rarely is). Never `skip` just to turn a red green. | ||
|
|
||
| **Verify locally — your PR gets no CI (step 6), so this is the only proof:** flake → re-run with | ||
| `--repeat-each` and confirm it's stable; skip → confirm it's skipped on the target config and | ||
| **still runs elsewhere**. Then run `npm run flint`. Record exactly what you ran — it goes in the | ||
| PR body. | ||
|
|
||
| ## 5. Pick a reviewer | ||
|
|
||
| No CODEOWNERS. Derive one from the touched file(s) — recent authors and frequent committers: | ||
|
|
||
| ```bash | ||
| git log --format='%an %ae' -n 20 -- <file> | ||
| git log --format='%an' -n 200 -- <file> | sort | uniq -c | sort -rn | ||
| ``` | ||
|
|
||
| Also skim recent merged PRs on those paths for who reviewed them. Pick someone with real recency | ||
| + ownership. Record it as a git trailer so it survives the handoff: | ||
|
|
||
| ``` | ||
| Suggested-reviewer: dgozman | ||
| ``` | ||
|
|
||
| ## 6. Commit and hand off (you never open the PR in CI) | ||
|
|
||
| You **never push or open a PR** in CI or locally. The harness does that for you, after you commit. | ||
|
|
||
| - **Exactly one commit** on the current branch (`CLAUDE.md` conventions; no | ||
| co-author / "generated with" trailers; never amend). | ||
| - **The commit message _is_ the PR** — the harness runs `gh pr create --fill`, mapping | ||
| subject→title and body→description. Write both in the Playwright bot voice | ||
| (`.github/workflows/bot-voice.md`) — verdict first, short, no slop. The | ||
| body must carry: the **DB evidence** (fail %, runs, bots) + any related issue; **what you | ||
| verified locally** and on which OS. | ||
| - **Nothing actionable?** Make no commit — the harness then skips the PR. | ||
|
|
||
| Report what you committed, the reviewer, and why. | ||
|
|
||
| ## Guardrails | ||
|
|
||
| - **One target, one commit.** Don't batch. | ||
| - **Scoped skips only** — narrow to the failing condition, never a whole file or browser matrix. | ||
| - **No unverified PRs** — if you couldn't run it locally, prefer a documented fixme. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this outside of skills, so that it's not accidentally used by the local dev agent?