fix(bench): repair the skopeo install and pin the competitor tools - #115
Merged
Conversation
skopeo renamed its module path to go.podman.io/skopeo at v1.23.0. The github.com path still serves tags, but their go.mod declares the new path, so the install fails on a module path mismatch. set -euo pipefail aborts the bootstrap there, before dregsy, regsync, the ocync build and the bench-proxy CA, and Terraform reports success because nothing waits on user-data. Instance bootstrap has been broken since that release. The three tools ocync is measured against were also installed with @latest, so every rebuild silently changed what the benchmark compared against. bench/results/{registry}.json is tracked for longitudinal comparison, which that undermines: a metric shift could come from a competitor release rather than from a change here. dregsy tags releases without the v prefix, so the module proxy serves only pseudo-versions for it. Its pin is the pseudo-version for release 0.5.2. dregsy and the credential helper report no usable version at runtime, one having no version flag and the other needing ldflags go install does not set, so the bootstrap log echoes their pins rather than probing them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark instance bootstrap is currently broken on
main, and has been since 2026-05-26. skopeo renamed its Go module path togo.podman.io/skopeoat v1.23.0. Thegithub.com/containers/skopeopath still serves tags, so it looks fine, but theirgo.moddeclares the new path and the install fails on a module path mismatch rather than a missing version:set -euo pipefailaborts user-data right there, so dregsy, regsync, the ocync and bench-proxy build, and the bench-proxy CA generation never run. Nothing waits on user-data, soterraform applystill reports success and the instance looks healthy. The failure only surfaces later as missing binaries.The three tools ocync is measured against were also installed with
@latest, so every instance rebuild silently changed what the benchmark compared against.bench/results/{registry}.jsonis tracked in git for longitudinal comparison, and that undermines it: a metric shift could come from a competitor release rather than from a change here. The Go toolchain immediately above these lines was already pinned, so it was only buying half the reproducibility it appeared to.Every pin is verified by fetching its
go.modfrom the module proxy and confirming the declared module matches the path being installed from. Checking that the version merely exists is not sufficient, and is exactly what was blind to the skopeo rename.dregsy needs explaining. It tags releases without the
vprefix, so the proxy cannot read them as versions and serves only pseudo-versions, meaning@latestwas never resolving to a release at all. Its pin isv0.0.0-20250317074629-e92e79a50145, whose commit is what the 0.5.2 tag points at.Two bootstrap log lines now echo the pinned version instead of probing the binary. dregsy has no version flag, so probing printed a usage dump, and the credential helper reports
developmentbecausego installsets none of the ldflags its Makefile uses. regsync and skopeo do report correctly and are left alone. Theexclude_graphdriver_devicemapperbuild tag is dropped, since that driver no longer exists in the storage dependency.Worth knowing before applying:
user_data_replace_on_change = true, so this recreates the instance. regsync moves from v0.11.3, which every archived record was measured on, to v0.11.5, so the first run after this is a new baseline rather than comparable to the numbers behind the published table.Two things this deliberately does not touch. The AMI still floats via
most_recent = true, so a version bump also rolls the OS image, andcheck_toolin the xtask runner ignores exit status, which is why 10 of 13 dregsy records in the archive store a flag error where a version belongs. Both are real and both are separate changes.