Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .claude/skills/playwright-test-results/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,20 @@ The snapshot may be missing the newest runs. To top it up locally, run `update`:
GITHUB_TOKEN=$(gh auth token) node utils/test-results-db/cli.ts update --lookback-days 3
```

Query it with the `duckdb` CLI (or any DuckDB client):
Query it through the bundled `@duckdb/node-api` binding — no separate DuckDB
install needed, it ships in `node_modules` after `npm ci`:

```bash
duckdb utils/test-results-db/test-results.duckdb "SELECT count(*) FROM test_results"
node --input-type=module -e '
import { DuckDBInstance } from "@duckdb/node-api";
const conn = await (await DuckDBInstance.create("utils/test-results-db/test-results.duckdb")).connect();
console.table((await conn.runAndReadAll(process.argv[1])).getRowObjectsJson());
' "SELECT count(*) FROM test_results"
```

Integer columns come back as strings (JSON-safe), so do ranking and filtering
in SQL, not in JS.

## Schema

Single table `test_results`, one row per test result (**one row per retry**).
Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/playwright-triage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ cannot-reproduce / not-a-bug; for a feature request or upstream/env issue: a sho
the condensed repro and be exhaustive about **what you ran** — the full matrix of browsers,
versions, and variations you tried, not just the one that worked — so the reader can trust the
verdict and skip re-checking. Call out any browser-specific divergence. Write it in the
[playwright-bot-voice](../playwright-bot-voice/SKILL.md) — maintainer voice, not AI-speak.
Playwright bot voice (`.github/workflows/bot-voice.md`, relative to the repo root) — maintainer voice, not AI-speak.

## Watch out

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
---
name: playwright-bot-voice
description: How the Playwright bot writes anything public — issue comments, PR descriptions and replies, release notes. Use whenever drafting text that will be posted under the bot's name on microsoft/playwright, and to keep agent-generated writing in the professional maintainer voice.
user_invocable: true
---

# Playwright Bot Voice

You're posting in public on `microsoft/playwright` **as the Playwright bot**. You don't need
Expand Down
111 changes: 111 additions & 0 deletions .github/workflows/fix-flakes-prompt.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this outside of skills, so that it's not accidentally used by the local dev agent?

Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Playwright: Fix a Flaky or Red Test

Turn CI test-results data into **one** concrete fix: pick a high-impact flaky-or-red test that is reproducible on your OS,
confirm nobody's on it, fix the root cause *or* scope a skip, pick a reviewer, and hand off a
single commit that becomes the PR. Fully autonomous — no approval stops.

## 1. Pick one target

Query the DB following the patterns in `.claude/skills/playwright-test-results/SKILL.md`.
Two families:

- **Cross-run flake** — the *final* verdict (after retries) flips between runs.
- **Consistent red** — `expected_status = 'passed'` yet failing in ~every run.

Rank by **impact**: fail %, run count (floor `runs >= 10` so it isn't a one-off), and how many
bots/PRs it disrupts — **not** by "has a tidy error message".
Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.
Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.

## 2. Check nobody's already on it

Search open PRs/issues for the test title words and file path before touching anything.
Also check any issue linked in the
test's `annotation`.
If an open PR already touches the test, **it's taken — drop it and go down your step-1 ranking**, re-checking each.
Only stop and report if the whole shortlist is covered.

## 3. Reproduce on this OS

Read the test and its `error_message`.

Build first (`npm run build`; watch is **not** running — if you touch generated-code files see
`CLAUDE.md`). Then reproduce, scoped to the failing target.

The DB `project_name` **is** the Playwright `--project`; `bot_name` encodes the OS. **Always
pass a `<file>:<line>` filter** — a bare run launches the whole suite. The browser test scripts
are project-locked, so use the one matching the failing bot:

| Failing target | Command |
|---|---|
| chromium | `npm run ctest -- <file>:<line>` |
| firefox | `npm run ftest -- <file>:<line>` |
| webkit | `npm run wtest -- <file>:<line>` |
| `tests/playwright-test/**` | `npm run ttest -- <file>:<line>` |
| `tests/mcp/**` | `npm run test-mcp -- --project=<name> <file>:<line>` |

Other suites (electron `etest`, etc.): see `package.json` scripts.
Add `--repeat-each=N` to force a flake's flip.

**You can only reproduce what this OS reaches.** A failure you can't reach here is itself
evidence it's environment-specific — a skip candidate (step 6), not a blind-fix.
Keep any OS handling keyed on the *current* OS, never hardcoded.
Broader OS coverage comes from different agent runs on other OSes, not from you.

## 4. Fix — root cause or scoped skip

- **Tractable → fix the source.** Usually a test-side race: a missing `await`, waiting on the
wrong signal, an under-specified locator, leaked state. Fix the test (or the product bug).
- **Engine/OS-specific, unreachable here, or genuinely hard → scope a skip**, narrowed to the
exact failing condition — never a whole file or browser:

```ts
it.fixme(browserName === 'webkit' && isLinux, 'https://github.com/microsoft/playwright/issues/NNNNN');
```

Link an issue if one exists or explain why the test is skipped. **Default to `fixme`** — it parks
the debt and stays greppable. Use `skip` only if reproduction shows the test is genuinely
mis-scoped for that config (`skip` claims "this failure is expected and correct," which a
flake-fighter rarely is). Never `skip` just to turn a red green.

**Verify locally — your PR gets no CI (step 6), so this is the only proof:** flake → re-run with
`--repeat-each` and confirm it's stable; skip → confirm it's skipped on the target config and
**still runs elsewhere**. Then run `npm run flint`. Record exactly what you ran — it goes in the
PR body.

## 5. Pick a reviewer

No CODEOWNERS. Derive one from the touched file(s) — recent authors and frequent committers:

```bash
git log --format='%an %ae' -n 20 -- <file>
git log --format='%an' -n 200 -- <file> | sort | uniq -c | sort -rn
```

Also skim recent merged PRs on those paths for who reviewed them. Pick someone with real recency
+ ownership. Record it as a git trailer so it survives the handoff:

```
Suggested-reviewer: dgozman
```

## 6. Commit and hand off (you never open the PR in CI)

You **never push or open a PR** in CI or locally. The harness does that for you, after you commit.

- **Exactly one commit** on the current branch (`CLAUDE.md` conventions; no
co-author / "generated with" trailers; never amend).
- **The commit message _is_ the PR** — the harness runs `gh pr create --fill`, mapping
subject→title and body→description. Write both in the Playwright bot voice
(`.github/workflows/bot-voice.md`) — verdict first, short, no slop. The
body must carry: the **DB evidence** (fail %, runs, bots) + any related issue; **what you
verified locally** and on which OS.
- **Nothing actionable?** Make no commit — the harness then skips the PR.

Report what you committed, the reviewer, and why.

## Guardrails

- **One target, one commit.** Don't batch.
- **Scoped skips only** — narrow to the failing condition, never a whole file or browser matrix.
- **No unverified PRs** — if you couldn't run it locally, prefer a documented fixme.
Loading
Loading