Skip to content

fix(bench): pin the AMI and record the binary that moves dregsy's bytes - #117

Merged
bryantbiggs merged 1 commit into
mainfrom
fix/bench-reproducibility
Aug 14, 2026
Merged

fix(bench): pin the AMI and record the binary that moves dregsy's bytes#117
bryantbiggs merged 1 commit into
mainfrom
fix/bench-reproducibility

Conversation

@bryantbiggs

@bryantbiggs bryantbiggs commented Aug 14, 2026

Copy link
Copy Markdown
Member

Two inputs still moved between otherwise identical instance rebuilds, so a benchmark shift could not be attributed to a change here.

The AMI was resolved with most_recent over the release prefix al2023-ami-2023.*-x86_64. AL2023 publishes several kernel variants under one release with near-identical creation timestamps, so that was not merely tracking the newest release, it was selecting between kernel 6.1, 6.12 and 6.18 arbitrarily. Kernel, glibc and NIC driver were all outside operator control on a network-bound benchmark. It now pins an exact image name, verified to match exactly one image in the region.

AWS deprecates AL2023 images after roughly three months and hides them from describe-images, so this pin eventually stops resolving and apply fails. That is intended. The failure is the prompt to move to a current image, rather than keep launching one with unpatched kernel and glibc updates, and the comment says so.

dregsy transfers nothing itself. Its generated config sets relay: skopeo, so skopeo moves every byte dregsy is credited with, and a skopeo bump shifted dregsy's numbers with nothing in the record to explain it. Relay binaries are now probed and recorded alongside the tool that delegates to them, and a relay failing its probe is an error, since it is as load-bearing as the tool itself.

Relay versions are kept in their own field rather than added to tool_versions. That map's keys drive the metric table columns, and a relay is never benchmarked, so reusing it would have rendered an empty skopeo column in every table. A test asserts the separation, kill-tested by feeding relay keys into the column list and confirming it fails.

CI is the gate for this one. Locally cargo fmt, clippy -D warnings, terraform fmt and terraform validate pass, and all 144 xtask tests pass. The eight testcontainers-backed integration tests in ocync-distribution currently fail on my machine with WaitContainer(StartupTimeout), which is container startup rather than an assertion, and this diff touches only bench/ and xtask/ so it cannot reach that crate. I have not proven the failure is environmental, so it wants a clean CI runner to confirm rather than my say-so.

@bryantbiggs
bryantbiggs force-pushed the fix/bench-reproducibility branch from c93e482 to 74533ee Compare August 14, 2026 00:15
Two inputs still moved between otherwise identical instance rebuilds, so a
metric shift could not be attributed to a change here.

The AMI was resolved with most_recent over a release prefix. AL2023 publishes
several kernel variants under one release with near-identical creation
timestamps, so that selected between kernels arbitrarily rather than merely
tracking the newest release, putting the kernel, glibc and NIC driver outside
operator control on a network-bound benchmark. It is now an exact name, which
matches a single image.

dregsy transfers nothing itself. Its generated config sets relay: skopeo, so
skopeo moves every byte dregsy is credited with, and a skopeo bump shifted
dregsy's numbers with nothing in the record to explain it. Relay binaries are
now probed and recorded alongside the tool that delegates to them. A relay
that fails its probe is an error, since it is as load-bearing as the tool.

Relay versions are kept in their own field rather than added to tool_versions,
whose keys drive the metric table columns. A relay is never benchmarked, so
reusing that map would render an empty column in every table.
@bryantbiggs
bryantbiggs force-pushed the fix/bench-reproducibility branch from 74533ee to 55906df Compare August 14, 2026 00:24
@bryantbiggs
bryantbiggs merged commit a3a89ff into main Aug 14, 2026
16 checks passed
@bryantbiggs
bryantbiggs deleted the fix/bench-reproducibility branch August 14, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant