fix(bench): pin the AMI and record the binary that moves dregsy's bytes - #117
Merged
Conversation
bryantbiggs
force-pushed
the
fix/bench-reproducibility
branch
from
August 14, 2026 00:15
c93e482 to
74533ee
Compare
Two inputs still moved between otherwise identical instance rebuilds, so a metric shift could not be attributed to a change here. The AMI was resolved with most_recent over a release prefix. AL2023 publishes several kernel variants under one release with near-identical creation timestamps, so that selected between kernels arbitrarily rather than merely tracking the newest release, putting the kernel, glibc and NIC driver outside operator control on a network-bound benchmark. It is now an exact name, which matches a single image. dregsy transfers nothing itself. Its generated config sets relay: skopeo, so skopeo moves every byte dregsy is credited with, and a skopeo bump shifted dregsy's numbers with nothing in the record to explain it. Relay binaries are now probed and recorded alongside the tool that delegates to them. A relay that fails its probe is an error, since it is as load-bearing as the tool. Relay versions are kept in their own field rather than added to tool_versions, whose keys drive the metric table columns. A relay is never benchmarked, so reusing that map would render an empty column in every table.
bryantbiggs
force-pushed
the
fix/bench-reproducibility
branch
from
August 14, 2026 00:24
74533ee to
55906df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two inputs still moved between otherwise identical instance rebuilds, so a benchmark shift could not be attributed to a change here.
The AMI was resolved with
most_recentover the release prefixal2023-ami-2023.*-x86_64. AL2023 publishes several kernel variants under one release with near-identical creation timestamps, so that was not merely tracking the newest release, it was selecting between kernel 6.1, 6.12 and 6.18 arbitrarily. Kernel, glibc and NIC driver were all outside operator control on a network-bound benchmark. It now pins an exact image name, verified to match exactly one image in the region.AWS deprecates AL2023 images after roughly three months and hides them from
describe-images, so this pin eventually stops resolving and apply fails. That is intended. The failure is the prompt to move to a current image, rather than keep launching one with unpatched kernel and glibc updates, and the comment says so.dregsy transfers nothing itself. Its generated config sets
relay: skopeo, so skopeo moves every byte dregsy is credited with, and a skopeo bump shifted dregsy's numbers with nothing in the record to explain it. Relay binaries are now probed and recorded alongside the tool that delegates to them, and a relay failing its probe is an error, since it is as load-bearing as the tool itself.Relay versions are kept in their own field rather than added to
tool_versions. That map's keys drive the metric table columns, and a relay is never benchmarked, so reusing it would have rendered an empty skopeo column in every table. A test asserts the separation, kill-tested by feeding relay keys into the column list and confirming it fails.CI is the gate for this one. Locally
cargo fmt,clippy -D warnings,terraform fmtandterraform validatepass, and all 144 xtask tests pass. The eight testcontainers-backed integration tests inocync-distributioncurrently fail on my machine withWaitContainer(StartupTimeout), which is container startup rather than an assertion, and this diff touches onlybench/andxtask/so it cannot reach that crate. I have not proven the failure is environmental, so it wants a clean CI runner to confirm rather than my say-so.