CI: combined timing stack - aggregate measurement against next#35358
Draft
valentinpalkovic wants to merge 24 commits into
Draft
CI: combined timing stack - aggregate measurement against next#35358valentinpalkovic wants to merge 24 commits into
valentinpalkovic wants to merge 24 commits into
Conversation
…d.ts generation Single rolldown pass per package with the tsgo declaration emitter replaces the per-entry child-process rollup fan-out. A hybrid resolver keeps the fast oxc resolution but resolves bare, non-external specifiers with TypeScript's module resolution, because oxc does not fall back to @types/* packages and ignores typesVersions (sxzz/rolldown-plugin-dts#130, oxc-resolver#549). Core d.ts generation drops from ~2.2 min to ~5 s. - --dts-bundler flag: rolldown-tsgo (default), rolldown (tsc emitter), rollup (legacy fallback) - --dts-resolver flag: hybrid (default), tsc, oxc - generate-source-files: enumerate globalized-runtime exports statically via rolldown - export QueryFunctions/ServiceRegistryApi/BoxOptions/LogMessageOptions used in public signatures - annotate checklistData and the react-syntax-highlighter deep import so declaration emit stays portable - delete the dead rollup dts() helper in scripts/utils/tools.ts
…mpatibility The rolldown-bundled d.ts renames chunk-level enum declarations (e.g. SupportedRenderer -> SupportedRenderer$1) when the plain name is taken by another chunk's import binding. TypeScript's cross-declaration enum compatibility is name-based, so in-repo code that mixes source-relative enum imports with dist-typed values stops type-checking. Import enums via the package specifier so both sides resolve to the same declaration in dev and production checks. Also export QueryFunctions and ServiceRegistryApi, which appear in inferred public signatures of dependent packages.
tsgo declaration emit hangs for minutes to hours on this package (likely @typescript-eslint's recursive types). Add a per-package dtsBundler override to the build config and use the tsc emitter there; it finishes in seconds.
Two clean tsgo builds are not byte-identical: type aliases flap between the alias and its expansion (e.g. TestProviderId vs string), which cascades into different chunk hash names, and one observed build wrote entry files whose chunk imports did not match the emitted chunk files (silently broken types, masked by skipLibCheck). The tsc emitter bundled by rolldown is byte-deterministic across builds, still ~14x faster than the rollup path (core: 2.2 min -> 9.1 s), and avoids the tsgo hang on eslint-plugin.
Emit each package's declarations with a single program.emit() call (noCheck, repo-root rootDir) and feed the pre-emitted tree to rolldown-plugin-dts via dtsInput. TypeScript's declaration emit is check-order-dependent, so letting the plugin emit per module in rolldown's concurrent load order produced byte-different output across runs (type aliases flapped between alias and expansion); the plugin's own build mode fixes the order but cannot handle this repo's cross-package source imports. Handwritten typings.d.ts inputs are mirrored into the emitted tree so side-effect imports resolve, and shared chunks are named chunk-[hash] because the default [name]-[hash] inherits an unstable base name. Two clean full production compiles now produce byte-identical declarations (SHA-256 over all 169 files); the full production check suite passes for all 44 projects.
The compile task's --parallel flag was still derived from the old NX agents experiment (amountOfVCPUs - 1 = 1), so the 43-project compile ran fully serially on the 8-vCPU xlarge executors of build--linux / build--windows. Peak RSS of the largest package build (core) is ~750MB, so 8 concurrent workers fit comfortably into the executor's 16GB.
Each of the 43 compile tasks paid ~0.7s for the yarn exec shim plus jiti's transform pass before doing any work. Node 22 runs .ts files natively via type stripping, which needs explicit .ts extensions on relative imports (sanctioned by the AGENTS.md jiti-to-node migration note). The build-config imports in entry-configs.ts pointed one directory too high (../../../code resolves above the repo root); jiti silently rescued them by falling back to CWD-relative resolution, which also forced @ts-ignore on every line. With the correct depth and explicit extensions TypeScript can resolve them, so the @ts-ignore lines are gone. MODULE_TYPELESS_PACKAGE_JSON warnings are disabled because the build-config files live in packages whose package.json intentionally has no "type" field; module-syntax sniffing still works, the flag only silences the log noise.
The 'Persisting to workspace' step is bytes-bound: CircleCI gzips the layer single-threaded at ~20MB/s, so the ~2.5GB of node_modules dominated the 139s step regardless of file count (a plain uncompressed tar measurably changed nothing). Pre-compressing with gzip -1 shrinks the payload ~3x, which cuts CircleCI's own compression, the upload, and the download in every one of ~100 downstream jobs. gzip is the only compressor available on all executor images; zstd is on none of them (verified via step diagnostics on cimg/node, playwright and machine images). Per-package node_modules folders only exist for packages with unhoistable dependencies, so the pack step filters them at runtime; the three root trees are passed to tar unconditionally so a missing one fails the job loudly.
The Playwright e2e-sandbox specs import scripts/utils/constants.ts and Playwright transpiles it to CommonJS, where import.meta is a syntax error - every sandbox dev/e2e job failed with 'exports is not defined in ES module scope'. Restore __dirname there and compute the repo root locally in the two build modules instead, which run as native ESM and never load under Playwright.
The check task looped ~44 workspaces sequentially, each spawning 'yarn exec jiti' (~0.7s shim + transform overhead per spawn, the same cost the compile task shed in the native-node migration). The whole TypeCheck code step ran 291s on a medium+ executor while using one of its vCPUs. The task now runs the per-package checks through a p-limit pool and launches check-package.ts with native node. The job executor moves to xlarge so eight checks run concurrently; the slowest packages (core, vue3, svelte, angular) are scheduled first so they don't stretch the tail of the run. Per-package output is buffered and only printed on failure so the parallel log stays readable. check-package.ts computes the repo root locally instead of importing scripts/utils/constants.ts, which intentionally uses __dirname (it must stay loadable from CJS-transpiled contexts like the Playwright specs) and therefore cannot load as native ESM. The scripts 'check' script moves from jiti to native node for the same reason the compile launcher did.
Every sandbox E2E pass ran with workers: 1 on CI, leaving the 3-4 vCPU Playwright executors mostly idle: 460-484s per dev-mode pass and 223-239s per static pass for the same spec files. The specs are already independent (fullyParallel was on, each test drives its own page), with one exception: change-detection.spec.ts writes to the sandbox's source files, and the resulting dev-server invalidations can reload other tests' pages mid-assertion. That spec moves to a chromium-mutating project that runs serially after the parallel chromium pass. CI now runs 3 workers per E2E step (overridable via PLAYWRIGHT_WORKERS), and the dev-mode jobs move from medium+ to large so the dev server and the workers don't starve each other - the shorter runtime more than pays for the class.
Same treatment the node_modules workspace got: the create jobs persisted the raw sandbox directory (mostly node_modules), paying CircleCI's single-threaded workspace compression on persist (31-61s per template) and again on every downstream attach. Pre-compressing with gzip -1 shrinks the payload ~3x, which cuts the create job's persist - the handoff every build/dev/e2e/vitest/chromatic job in the template's flow waits on - and the attach in each of those jobs. Downstream jobs unpack the per-template archive right after the node_modules unpack; the Windows sandbox jobs (daily workflow) get the same step, since they consume the Linux-created sandbox from the shared workspace.
The first run of the sandbox tarball broke the dev jobs of every
change-detection template two ways:
- tar extraction as root (the Playwright image) preserved the create job's
uid, unlike attach_workspace which materializes files as the current
user. git then refused to operate on the sandbox repo ('dubious
ownership'), breaking the change-detection feature and its E2E tests.
--no-same-owner restores attach_workspace semantics.
- the default gnu tar format truncates mtimes to whole seconds, which
invalidated webpack's filesystem-cache snapshots and turned the next-js
webpack dev start into a full cold compile. The e2e task's 1s dev
readiness probe then timed out, spawned a second dev server on the same
port, and crashed with EADDRINUSE when that server finished compiling.
posix (pax) format keeps sub-second mtimes.
The dev readiness probe also gets a CI-sized timeout: when a server is
already listening but still compiling its first preview bundle, waiting is
correct and spawning a second server never is. Locally the probe stays at
1s (a dead port still fails fast with ECONNREFUSED either way).
ESLint (177s), Knip (109s) and format check (79s) each paid ~50s of spin-up, checkout and workspace restore per pipeline - and the format-check job additionally ran its own 46s yarn install because it never attached the workspace. None of the three is near the workflow critical path, so they now run as steps of one 'Static checks' job on the workspace, format check first for the fastest failure signal. The standalone fmt job remains for the docs workflow, which has no build job to restore a workspace from.
--skip-nx-cache on CI was a leftover of the concluded NX Cloud agents experiment (per its own comment) and fully disconnected the run from Nx Cloud - the remote cache was neither read nor written. Removing it lets unchanged packages restore their dist from the remote cache instead of recompiling on both build--linux and build--windows. Cache correctness rests on the d.ts determinism guarantee from the rolldown-dts work (byte-identical outputs across clean runs) and on the compile target's inputs: sharedGlobals covers scripts/**/* (the whole build toolchain) and code/tsconfig.json, production covers each project's sources. Untrusted runs without an Nx Cloud token degrade to a warning and an uncached compile.
The standalone (build) job per template was ~90s of which ~55s was fixed
setup (spin-up, checkout, workspace attach and unpack) around a ~25-45s
build - and it sat between (create) and the e2e/chromatic/vitest jobs on
the workflow critical path, adding a full job hop plus a workspace
round-trip to those chains.
The static build now runs at the end of the create job, before the sandbox
is packed, so storybook-static travels inside the existing sandbox tarball
and downstream jobs need no extra restore step. e2e, chromatic, vitest and
test-runner depend on create directly.
The Windows sandbox jobs and the daily test-runner job referenced flow
jobs by array index; they now use named references, since the build job no
longer exists. saveBench('build') records identical data - it runs inside
'yarn task build' regardless of which CI job hosts it, and no job in the
generated workflows consumed the bench task output.
Even with parallel Playwright workers, the dev-mode E2E jobs are the workflow's wall-clock tail (300-377s each, starting only after their template's create job). CircleCI parallelism 2 halves the spec list per container via the existing 'circleci tests run' scaffolding - the --index/--total flags were already in place, hardcoded to a single node. Each shard starts its own dev server; change-detection.spec.ts lands on exactly one shard, where the serial chromium-mutating project isolation still applies. The smoke-test dev variant (bench templates) stays unsharded - it runs no Playwright pass.
Playwright runs dependency projects unfiltered: on a sharded dev job whose file subset contained change-detection.spec.ts, the chromium-mutating project's dependency on chromium re-ran the entire chromium suite (105 tests instead of the shard's 41), erasing the sharding win on exactly one of the two containers. The e2e task now runs two sequential playwright invocations - the parallel chromium pass, then the serial chromium-mutating pass - which preserves the isolation ordering while respecting the shard's file filter. --pass-with-no-tests covers shards whose subset has no match for one of the projects, and the mutating pass writes its junit to a -mutating suffixed file so the two reports don't overwrite each other. chromium-mutating keeps only the cheap 'setup' project as a dependency.
Four independent trims, measured on a timestamped local reproduction of angular-vite sandbox creation (107s -> 56s) and a phase decomposition of the CI create jobs (83-106s 'Create Sandbox' step): - Yarn setup becomes two file writes. installYarn2 ran 'yarn set version berry' plus chained 'yarn config set' calls: ~12 yarn boots and two network downloads (corepack's bootstrap yarn, then the latest Berry) per sandbox. The sandbox now vendors the repository's own pinned yarn release via yarnPath and writes .yarnrc.yml directly. packageManager is pinned in the sandbox package.json to the same version: without the field, corepack auto-pins its classic bootstrap yarn, and JsPackageManagerFactory would read that field and misclassify the sandbox as Yarn 1 (caught by the local reproduction; installs kept working through yarnPath delegation while the CLI ran classic-syntax commands). - The final 'yarn install' no longer wipes node_modules first. The wipe forced a full re-extraction of the tree (~10-25s on CI) that yarn's incremental install makes redundant. - Extra sandbox deps carry explicit version ranges, so addExtraDependencies does not probe the registry (a subprocess each) to resolve 'latest' for packages whose major we already know - and sandbox contents stop drifting when a new major is published. - The executor images move to cimg/node:22.22.3, matching .nvmrc. That satisfies Angular's minimum Node version, so the per-job 'node/install' step (~9s on every angular create/dev job) is gone; the create jobs also pre-warm the npx cache for the pinned gitpick version the CLI spawns, so the template download does not pay a cold npx install. Rejected while implementing: passing skip-install to the init CLI step (would fold init's install into the final one, ~15-25s) silently no-ops all addon configuration, because postinstallAddon resolves each addon's postinstall hook from the sandbox's node_modules and returns quietly when resolution fails - addon-vitest's setup would disappear from sandboxes.
The readiness probe fetched /iframe.html and re-ran the dev task on any failure. But the fetch can fail while a dev server owns the port - a cold compile outlasting the timeout, or the server resetting connections before its middleware is up (seen on a sveltekit shard: the probe got a reset, the task spawned a second server, and that server crashed the job with EADDRINUSE once its own compile finished). A held port now counts as ready via a TCP-connect fallback; every consumer of the dev task performs its own HTTP wait before using the server, so this cannot mask a dead server. This also replaces the earlier CI-only 180s probe timeout with behavior that is correct locally too.
Sandbox creation spawned ~24 package-manager subprocesses just to resolve versions, measured via a timestamped local reproduction of angular-vite sandbox creation and confirmed in the CI create-job logs: - latestVersion() probed the registry for every storybook package even though the CLI ships the authoritative version map (versions.ts) and the sandbox registry (verdaccio) serves exactly those versions. Inside sandbox creation (IN_STORYBOOK_SANDBOX, set by the monorepo task runner and never by real user installs) the probe is now short-circuited to the versions map. On CI the resulting specifiers are byte-identical; in link-mode local sandboxes storybook packages get ^version instead of an exact pin, where the linked workspace overrides resolution anyway. - addDependencies(skipInstall) resolved a version for every dependency serially - including dependencies whose specifier already carried one, discarding the result. Lookups now only happen for unversioned specifiers and run concurrently, with insertion order kept deterministic. This also speeds up real 'storybook init' runs. - getInstalledVersion() spawned a full dependency-graph walk per package (yarn info --recursive), which exits 1 with usage output for packages that are not installed at all - addon-vitest probes vitest, msw and coverage packages this way on every sandbox. The module's own package.json is now resolved first (a plain fs read, PnP included), and the graph walk only runs as a fallback for declared dependencies that don't resolve directly (e.g. pnpm virtual store layouts). - The 'sb repro' command probed npm for latest/next before downloading the template; inside sandbox creation those two roundtrips only feed the 'you are behind' warning copy and are skipped. The proxy unit tests asserting registry-probe behavior now stub IN_STORYBOOK_SANDBOX off explicitly - the repo-level .env marks all vitest runs as sandbox context - and a new test pins the short-circuit contract. Measured locally (angular-vite, full creation): 24 -> 8 version-probe spawns; 56s -> 48s on top of the CI-side trims, 107s cumulative baseline.
valentinpalkovic
added
build
Contributor
|
|
View your CI Pipeline Execution ↗ for commit fd38349
💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗ ☁️ Nx Cloud last updated this comment at |
Package BenchmarksCommit: The following packages have significant changes to their size or dependencies:
|
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 72 | 72 | 0 |
| Self size | 21.94 MB | 21.11 MB | 🎉 -831 KB 🎉 |
| Dependency size | 36.44 MB | 36.44 MB | 0 B |
| Bundle Size Analyzer | Link | Link |
@storybook/angular-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 34 | 34 | 0 |
| Self size | 22.41 MB | 2.40 MB | 🎉 -20.02 MB 🎉 |
| Dependency size | 22.54 MB | 42.56 MB | 🚨 +20.02 MB 🚨 |
| Bundle Size Analyzer | Link | Link |
@storybook/ember
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 187 | 187 | 0 |
| Self size | 15 KB | 13 KB | 🎉 -2 KB 🎉 |
| Dependency size | 30.69 MB | 30.69 MB | 🚨 +890 B 🚨 |
| Bundle Size Analyzer | Link | Link |
@storybook/nextjs
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 533 | 533 | 0 |
| Self size | 662 KB | 640 KB | 🎉 -23 KB 🎉 |
| Dependency size | 62.06 MB | 62.03 MB | 🎉 -34 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/nextjs-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 93 | 93 | 0 |
| Self size | 1.39 MB | 1.37 MB | 🎉 -22 KB 🎉 |
| Dependency size | 23.99 MB | 23.95 MB | 🎉 -37 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/react-native-web-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 122 | 122 | 0 |
| Self size | 30 KB | 29 KB | 🎉 -988 B 🎉 |
| Dependency size | 25.06 MB | 25.02 MB | 🎉 -37 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/react-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 83 | 83 | 0 |
| Self size | 36 KB | 32 KB | 🎉 -4 KB 🎉 |
| Dependency size | 21.76 MB | 21.73 MB | 🎉 -33 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/react-webpack5
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 274 | 274 | 0 |
| Self size | 23 KB | 23 KB | 🎉 -789 B 🎉 |
| Dependency size | 47.84 MB | 47.81 MB | 🎉 -34 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/tanstack-react
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 84 | 84 | 0 |
| Self size | 110 KB | 104 KB | 🎉 -5 KB 🎉 |
| Dependency size | 21.80 MB | 21.76 MB | 🎉 -37 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/cli
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 204 | 204 | 0 |
| Self size | 821 KB | 821 KB | 🚨 +67 B 🚨 |
| Dependency size | 91.79 MB | 90.96 MB | 🎉 -829 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/codemod
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 197 | 197 | 0 |
| Self size | 32 KB | 32 KB | 🚨 +151 B 🚨 |
| Dependency size | 90.27 MB | 89.44 MB | 🎉 -829 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
create-storybook
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 73 | 73 | 0 |
| Self size | 1.09 MB | 1.09 MB | 🚨 +66 B 🚨 |
| Dependency size | 58.38 MB | 57.55 MB | 🎉 -831 KB 🎉 |
| Bundle Size Analyzer | node | node |
@storybook/react
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 59 | 59 | 0 |
| Self size | 1.52 MB | 1.49 MB | 🎉 -35 KB 🎉 |
| Dependency size | 12.43 MB | 12.43 MB | 🚨 +169 B 🚨 |
| Bundle Size Analyzer | Link | Link |
Contributor
Author
Aggregate results: 82/82 green, wall 13.6 minWorkflow
Sanity gates on this run:
Critical path now: The stack rebased onto |
valentinpalkovic
added
ci:merged
Contributor
Author
Merged + daily workflows measured on this branchLabel-triggered both remaining workflows on this branch to complete the picture. Credits are CircleCI's own
Combined with the normal-workflow run above (18.1 -> 13.6 min wall, 5,575 -> 4,945 credits), projecting the measured per-workflow ratios onto the last 30 days of usage (insights API: normal 2.39M, merged 0.73M, daily 1.44M credits):
The daily run also revalidates the Windows tarball-unpack path and the full template matrix: 187/187 green including Windows, vitest, and test-runner jobs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurement PR - not the merge unit. This branch is the entire CI-timing stack rebased onto current
next(clean rebase, zero conflicts;yarn install --immutableand the scripts typecheck pass locally). Its purpose is to answer one question: how do all the fixes add up againstnext, in one pipeline run?The individual PRs remain the review/merge units:
Baselines (same CircleCI v1.1 API methodology throughout)
a65d3769)069b53d0)Results from this branch's run will be posted as a comment when the workflow completes.
🤖 Generated with Claude Code
https://claude.ai/code/session_017eZuBpKfFYbF52fZiHmKMo