Skip to content

fix(bench): repair the skopeo install and pin the competitor tools - #115

Merged
bryantbiggs merged 1 commit into
mainfrom
chore/pin-bench-tools
Aug 13, 2026
Merged

fix(bench): repair the skopeo install and pin the competitor tools#115
bryantbiggs merged 1 commit into
mainfrom
chore/pin-bench-tools

Conversation

@bryantbiggs

Copy link
Copy Markdown
Member

Benchmark instance bootstrap is currently broken on main, and has been since 2026-05-26. skopeo renamed its Go module path to go.podman.io/skopeo at v1.23.0. The github.com/containers/skopeo path still serves tags, so it looks fine, but their go.mod declares the new path and the install fails on a module path mismatch rather than a missing version:

module declares its path as: go.podman.io/skopeo
        but was required as: github.com/containers/skopeo

set -euo pipefail aborts user-data right there, so dregsy, regsync, the ocync and bench-proxy build, and the bench-proxy CA generation never run. Nothing waits on user-data, so terraform apply still reports success and the instance looks healthy. The failure only surfaces later as missing binaries.

The three tools ocync is measured against were also installed with @latest, so every instance rebuild silently changed what the benchmark compared against. bench/results/{registry}.json is tracked in git for longitudinal comparison, and that undermines it: a metric shift could come from a competitor release rather than from a change here. The Go toolchain immediately above these lines was already pinned, so it was only buying half the reproducibility it appeared to.

  • docker-credential-ecr-login v0.12.0
  • skopeo v1.24.0, from the new module path
  • regsync v0.11.5
  • dregsy release 0.5.2

Every pin is verified by fetching its go.mod from the module proxy and confirming the declared module matches the path being installed from. Checking that the version merely exists is not sufficient, and is exactly what was blind to the skopeo rename.

dregsy needs explaining. It tags releases without the v prefix, so the proxy cannot read them as versions and serves only pseudo-versions, meaning @latest was never resolving to a release at all. Its pin is v0.0.0-20250317074629-e92e79a50145, whose commit is what the 0.5.2 tag points at.

Two bootstrap log lines now echo the pinned version instead of probing the binary. dregsy has no version flag, so probing printed a usage dump, and the credential helper reports development because go install sets none of the ldflags its Makefile uses. regsync and skopeo do report correctly and are left alone. The exclude_graphdriver_devicemapper build tag is dropped, since that driver no longer exists in the storage dependency.

Worth knowing before applying: user_data_replace_on_change = true, so this recreates the instance. regsync moves from v0.11.3, which every archived record was measured on, to v0.11.5, so the first run after this is a new baseline rather than comparable to the numbers behind the published table.

Two things this deliberately does not touch. The AMI still floats via most_recent = true, so a version bump also rolls the OS image, and check_tool in the xtask runner ignores exit status, which is why 10 of 13 dregsy records in the archive store a flag error where a version belongs. Both are real and both are separate changes.

skopeo renamed its module path to go.podman.io/skopeo at v1.23.0. The
github.com path still serves tags, but their go.mod declares the new path,
so the install fails on a module path mismatch. set -euo pipefail aborts
the bootstrap there, before dregsy, regsync, the ocync build and the
bench-proxy CA, and Terraform reports success because nothing waits on
user-data. Instance bootstrap has been broken since that release.

The three tools ocync is measured against were also installed with @latest,
so every rebuild silently changed what the benchmark compared against.
bench/results/{registry}.json is tracked for longitudinal comparison, which
that undermines: a metric shift could come from a competitor release rather
than from a change here.

dregsy tags releases without the v prefix, so the module proxy serves only
pseudo-versions for it. Its pin is the pseudo-version for release 0.5.2.
dregsy and the credential helper report no usable version at runtime, one
having no version flag and the other needing ldflags go install does not
set, so the bootstrap log echoes their pins rather than probing them.
@bryantbiggs
bryantbiggs merged commit 79cf3f1 into main Aug 13, 2026
16 checks passed
@bryantbiggs
bryantbiggs deleted the chore/pin-bench-tools branch August 13, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant