Skip to content

Latest commit

 

History

History
135 lines (111 loc) · 6.61 KB

README.md

File metadata and controls

135 lines (111 loc) · 6.61 KB

E2E benchmark dashboard

This directory contains the static dashboard for versioned Snapshot E2E benchmark history. Vite and TypeScript are build-time tools only: GitHub Pages serves the generated HTML, CSS, JavaScript, and benchmark index without a Node.js server or runtime CDN dependency.

The UI discovers suites, cases, and measurements from the history data. It does not contain a framework-specific result schema, so future E2E suites can publish the same generic benchmark envelope and appear without new parsing code.

Local development

The Node.js version CI uses is pinned in .nvmrc (nvm use picks it up). The scripts need Node.js 22.18 or newer because they run TypeScript natively; scripts/check-node.mjs fails early with a clear message on older versions. Install dependencies and start Vite from this directory:

npm ci
npm run dev

The local server reads the small fixture history under public/index. The fixture is generated by the history writer (snapshot_e2e.benchmark_history rebuild) so it carries the exact manifest, comparisonKey, and record shape that production publishes. Run all local dashboard checks with:

npm run typecheck
npm test
npm run validate-data -- --strict
npm exec playwright -- install chromium
npm run test:browser

npm run test:browser first builds the production bundle (npm run build:production, which excludes the fixture data), copies the fixture index into dist/index, and serves dist/ with vite preview, so the browser tests exercise the same artifact that GitHub Pages deploys. One test asserts that every request during load stays on the test origin.

To validate a checked-out durable history branch instead of the fixture data:

npm run validate-data -- /path/to/e2e-benchmark-history/index

Without --strict the validator reports invalid records as warnings (as GitHub Actions annotations when it runs in CI) and only fails on a malformed manifest or a record count that does not match it. The dashboard skips invalid records at load time and lists them, with their chunk path and line, under the status line. --strict (used by ci.yml for the fixture) fails on any invalid record.

History JSON is treated as untrusted input. TypeScript checks the dashboard at build time, while runtime parsing validates every manifest and NDJSON record, including RFC3339 timestamps. Values are rendered through DOM text properties rather than raw HTML.

Loading and comparisons

The dashboard fetches the manifest, then only the monthly chunks that overlap the default 90-day window, in parallel. Older months load on demand when a wider date range is selected. The status line shows the newest published result and flags history older than three days as stale.

Comparisons use the comparisonKey emitted by the history writer on every index line; records without one fall back to the same derivation implemented in src/data.ts, which src/data.test.ts checks against the shared fixture in e2e/tests/data/comparison-key.json. Every measurement that is not complete is listed as an explicit gap under its chart, including incomplete measurements on otherwise passing runs. Clicking a chart point opens the run details with the previous-run and seven-run-median comparison, the commit, and links to the GitHub Actions run and commit.

Colors

src/colors.ts owns every series color. Framework cases have fixed colors (vLLM teal, SGLang blue, TensorRT-LLM amber) and unknown cases pick from a fallback list by a stable hash of the case name, so a case keeps its color regardless of which other cases are visible.

Stage segments in the stacked run chart are colored by measurement name, never by position in the current selection. Every measurement the agent and the framework test emit today (26 names) has its own entry in a fixed table; only a name missing from the table falls back to a stable hash of the name into three reserved colors. The eight muted stage colors behind the table are split in half: checkpoint-side measurements take the cool stops (teal, blue, green, slate) and restore-side measurements take the warm stops (rust, plum, purple, ochre), while infrastructure measurements (image pulls, agent-complete-to-traffic, total test time) are neutral grays, so a stacked bar reads as checkpoint half, restore half, and overhead at a glance. Sub-phases are lighter or darker shades of the same stops. The primary stages are at least ΔE 8 apart in OKLab and every pair within a family at least ΔE 7 apart, so the legend and tooltips are the secondary cue for neighbours in the same half. A measurement name that is not in the table takes one of three reserved muted colors that no known measurement uses. src/colors.test.ts enforces these guarantees; add new agent phases to the table there rather than relying on the reserved colors.

GitHub Pages deployment

The E2E Benchmark Dashboard workflow builds source from main, checks out the data-only e2e-benchmark-history branch through the shared .github/actions/fetch-benchmark-history action, validates it, and copies its index/ directory into the static artifact. Fixture data is excluded from production builds. The workflow deploys after dashboard changes and after every scheduled framework workflow, including runs containing failed or partial benchmarks. Deployments are queued rather than cancelled so an in-flight Pages deployment is never interrupted. Before the first scheduled run publishes history, the site deploys with an empty manifest and explains that no results exist yet.

Scheduled main results update the root dashboard and durable history. A pull request mirror or manual framework run instead gets a temporary URL under /previews/pr-<number>/ or /previews/run-<id>/. Its validated result is overlaid on a read-only copy of nightly history, so the graph can be inspected before merge without changing the baseline. Preview indexes live on the separate generated branch e2e-benchmark-previews, expire after 14 days, and are clearly labeled in the UI. The Pages deployment job summary contains the final preview link.

The privileged preview job is a deliberate trust boundary: GitHub runs its workflow and Python publisher from main, never from the source branch. It accepts only size-limited benchmark JSON whose run identity matches the triggering workflow, and it never executes artifact content.

A repository administrator must enable Pages once in Settings → Pages → Build and deployment → Source → GitHub Actions. No Node.js process runs after the artifact is deployed.