Add RWX CI for the Zig compiler (Linux x64 + arm64), running in parallel with GHA#10043
Draft
djpenka wants to merge 9 commits into
Draft
Add RWX CI for the Zig compiler (Linux x64 + arm64), running in parallel with GHA#10043djpenka wants to merge 9 commits into
djpenka wants to merge 9 commits into
Conversation
Stand up an RWX-based Linux CI pipeline that runs in parallel with GitHub Actions (non-blocking), reproducing ci_zig's Linux test matrix plus cross-compilation coverage, with Captain for per-test results and retry. - .rwx/ci.yml + .rwx/ci-arm64.yml: build-once anchor + consumer-leaf DAG (x86_64 with an embedded arm64 run), positive input filters (docs-only change -> cache HIT), a single network boundary for deps (parallel fetch of the lazy cross-compile bundles), a 6-target compile-only cross matrix, the plan.md gate, and GitHub PR triggers. - .captain/config.yml: rwx-v1-json suites for unit/eval/cli/lsp/platforms/ playground (inline per-test results + targeted retry + flake detection). - build.zig, src/build/*, and the test runners: a gated rwx-v1-json reporter (-Droc-test-runner / -Droc-test-rwx-out), a shared test_harness, the cross-agent host-tools caching fix (cpu_model = .baseline), and the playground test optimize set to ReleaseSmall to match GHA.
…che stability) Host-arch builds left cpu_model unset for aarch64, which resolves to the building machine's NATIVE CPU (Zig Query.cpu_model defaults to .determined_by_arch_os → native on a host build). x86_64 already pinned x86_64_v3, but aarch64 did not. On CI this makes the compile-cache key agent-specific: RWX runners vary by CPU model, so test binaries built by a build anchor never cache-hit on a differently-provisioned leaf agent, and the leaf rebuilds the whole suite (the Debug leaf OOMs; the ReleaseFast leaf hits the 10-min timeout). Same class of bug as the existing hostToolsTarget fix. Centralize the policy in a stableHostCpuModel(arch) helper (x86_64_v3 for x86_64, baseline otherwise) and apply it at all host-arch target sites: the default target, getReleaseTargetQuery, nativeSharedArchiveTarget, and the fx/http platform host libs. Note: this changes the DEFAULT aarch64 build (including local dev) to baseline ARMv8 instead of native — consistent with how x86_64 already pins x86_64_v3, and it makes distributed arm binaries deterministic. Pass -Dcpu=native for a machine-tuned local build.
…mpute) Rust: the rebase pulled in RustGlue regression tests (run-test-cli, suite cli-platforms) that shell out to rustc. GitHub runners ship rustc for free; the rwx/base image does not, so platforms failed with error.FileNotFound. Add a rust-lang/install 1.0.7 task (rust 1.83.0) on both legs, used only by the platforms task (cli-llvm filters glue out). Sizing: every heavy task ran on a 32c agent, but roc's builds are serial-bound (the roc binary is the long pole) so most barely used the cores. RWX bills per vCPU-second with no per-GB charge, so cores are the cost lever. Right-size each task to its real CPU/memory footprint: serial builds (valgrind) and off-critical-path builds (cross-compile, build-ci) drop to 4-8c; tracy to 8c; build-release-arm64 to 16c. build-release-x64 is kept at 32c — it is genuinely parallel, and shrinking it cost both time and money. Memory class (CO/GP/MO) follows each task's measured peak at its chosen core count. Result (same-warmth A/B): full-run compute -47% ($6.32 -> $3.38/run), heavy tasks -51%, wall-clock flat (681s -> ~699s). Detail in tmp/rwx-migration/agent-sizing-experiment.md.
When a test hard-aborts (e.g. a std.debug.panic in a Debug build), Zig's
build runner re-spawns the test binary to continue the remaining tests,
replaying run_test against its cached metadata *without* re-sending
query_test_metadata (std.Build.Step.Run only queries when its metadata is
null). Our partitioned runner only populates selected_indices on
query_test_metadata, so the re-spawned process indexed an empty slice and
panicked ("index out of bounds: index N, len 0") on every re-spawn.
That secondary panic masked the genuine failure and broke Zig's
crash-recovery: it couldn't continue past the aborting test, so the rest of
that binary's tests were dropped from the results (observed ~140 tests
missing from the reported total).
Rebuild selected_indices lazily in the run_test handler when it's empty.
buildSelectedIndices is deterministic for a given config, so the index Zig
replays from its cached metadata still maps correctly. The crash is now
isolated to the offending test (reported as crashed) and the rest of the
suite completes; a defensive bounds check replaces the raw slice panic.
The stock Zig runner is immune because it indexes builtin.test_functions
directly; only our partition indirection needed this.
Warm tool-cache restores saved ~0 compile time on real PRs: Zig caches whole compilation units, the roc binary embeds ~all of src/, and builtin_compiler imports the compiler core, so any source change reruns the builtins bake and cascades into every binary embedding compiled_builtins. Meanwhile the caches cost 6.5-13 GB layer downloads per anchor, split the inherited filesystem so every leaf pulled ~24.6 GB instead of ~14 GB when warm, and expired on a 48h TTL anyway. Dropped from all compile tasks; the vault stays for the tool caches packages declare internally. The one genuinely cacheable piece was the fixed prelude: compiling build.zig itself into the build runner (~94s at 4 cores), paid separately by every task that invokes `zig build` without an anchor upstream. New zig-build-runner-* tasks compile it once via the configure-only `zig build --list-steps` into $HOME/.zig-cache; consumers inherit it through use:. Their filter is exactly the runner's compile closure (build.zig, build.zig.zon, src/build, src/target), so source-only PRs content-cache HIT and get the prebuilt runner for free - no TTL, and it works on cold runs, which the tool cache never did. Also exclude test/snapshots from the anchor filters (negation): snapshot .md files are runtime data for the snapshot tool, not a compile input, so snapshot-only PRs now content-HIT every anchor instead of recompiling the world. Leaves keep them (snapshot_tool's unit test reads the dir at runtime). Validated across three full runs (identical code): valgrind anchor (crit path) 637s -> 565s, build-ci-x64 503s -> 433s, runner phase 3-19s on all anchors, unit-debug test count identical pre/post filter change (3207).
The old gates/git-lints split claimed to keep gates cacheable, but gates also used code-git (tidy + zig-lints need a worktree) so both tasks missed every commit. Invert the split by what actually needs git: - gates: the non-git checks (test-wiring, builtin-format, fx-coverage, serialization-sizes) on the git-less clone with the pos-build filter, so docs- and snapshot-only changes cache-hit instead of paying the ~150s re-check. - git-gates: the worktree-dependent checks (zig-lints, tidy, git-lints, plan.md gate) on code-git; re-runs every commit by design but is only ~15s of exec. Also move the shared YAML anchors (pos-build, pos-leaf, pos-cli, test-agent) into top-level aliases: blocks in both files, compose pos-cli from pos-leaf via array flattening (deduping the two hand-copied filter lists on the arm64 leg), and trim stale comments. Validated on RWX: gates+git-gates run 7cdd4352 green with every check's success output present in the logs; cli-llvm run 5deb2cd2 green with the flattened pos-cli filter (fixture imports found).
Comment-only cleanup across both files (the detailed rationale lives in the migration notes / git history), plus one real change: tracy-check's filter now reuses the pos-build alias via array flattening instead of an inline copy — resolves to the identical path set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add RWX CI for the Zig compiler (Linux x64 + arm64), running in parallel with GHA
This adds an RWX CI pipeline covering the new Zig compiler's Linux CI (x86_64 + arm64), running alongside the existing GitHub Actions workflows. Nothing is removed and the new check is not required — the idea is to let it soak in parallel so we can compare reliability/speed before deciding anything further. macOS/Windows (and the mac/win-host-coupled jobs like
roc-cross-compile's on-target runs) stay on GHA regardless.What's in the PR
.rwx/ci.yml+.rwx/ci-arm64.yml— the pipeline. Same coverage as GHA's Linux CI: debug/release build anchors, module unit tests (debug + release), snapshots (with the zero-driftgit diffcheck), eval/CLI LLVM suites, platform tests, LSP integration, valgrind, fuzz repro, cross-compile compile-only checks (all 6 targets), tracy compile-check, static-link assertion, parser coverage (kcov, arm64), and the lint/tidy/format gates including theplan.mdcheck.The main structural difference from GHA: tasks declare positive input filters, so RWX's content-based cache skips whatever a change can't affect. A docs-only change cache-hits essentially the whole graph; a snapshot-only PR (new/edited
.mdtests, nosrcchange) skips all compiles and only re-runs the tasks that actually read snapshots..captain/config.yml— Captain suite definitions. This is what gives per-test results inline in the run UI, flake detection, and "retry only the failed tests" instead of re-running a whole job.build.zig— two things:-Droc-test-runner,-Droc-test-rwx-out,-Droc-test-files, …). All default off: a plain localzig build teststill uses Zig's stock test runner, so nothing changes for contributors. CI turns them on so test binaries report per-test JSON and support partitioned/targeted re-runs.stableHostCpuModel: host-arch builds now pin a fixed CPU model (x86_64_v3on x64 — which was already the effective default — andbaselineon aarch64) instead of resolving to the build machine's native CPU. This is needed for cross-machine build caching (otherwise every agent gets a different cache key and recompiles the world), but note it changes the default aarch64 build, including local dev builds, from native-CPU to baseline.test/zig_test_runner.zig+ the parallel runners — the previously-dormant custom runner is wired up (behind the flags above) with an rwx-v1-json reporter and partition-aware test selection. This also fixes a real pre-existing bug in the runner: it didn't survive Zig's crash-recovery re-spawn (the build runner re-spawns a test binary after a hard crash to run the remaining tests), which silently dropped ~140 tests from the reported totals whenever a test crashed.Validation
Full runs of the assembled pipeline are green end-to-end on a scratch RWX org (both legs, all tasks), with test-count parity checked against the GHA runs (e.g. the unit suite reports the same totals). Along the way the pipeline surfaced two pre-existing
mainissues (a deterministicroc_alloc calledabort injson_decoder_platform_teston both arches, and a valgrind uninitialised-value hit inCoreCtx.osCanonicalize) — I can file those separately if they're still reproducible on current main.Operational notes
main, readable by fork PRs so they get cache hits but can't poison it).What this needs from the roc team
roc-langorg (I'll coordinate — this is what triggers runs on PRs).Happy to walk through any of it, split the
build.zigCPU-pin change into its own PR, or adjust scope.