Skip to content

fix(bench): stop recording probe errors as tool versions - #116

Merged
bryantbiggs merged 1 commit into
mainfrom
fix/bench-version-probe
Aug 13, 2026
Merged

fix(bench): stop recording probe errors as tool versions#116
bryantbiggs merged 1 commit into
mainfrom
fix/bench-version-probe

Conversation

@bryantbiggs

Copy link
Copy Markdown
Member

check_tool never inspected the exit status and fell back to stderr, so a tool that exits non-zero from its version probe had the error text stored as its version. dregsy has no version flag at all, so 10 of its 13 records in bench/results/ecr.json hold a Go flag error where a version belongs:

"version": "time=\"2026-04-26T13:17:46Z\" level=error msg=\"flag provided but not defined: -version\""

That defeats what the archive is tracked in git for. A run record that cannot say which dregsy produced a number cannot attribute a shift to a version change.

Tolerating a failed probe is specific to the tool that has no version to report. check_tool is the only gate that runs before a benchmark starts, so for a tool that does report one, a failed probe is now an error rather than a quietly absent version -- otherwise a broken ocync or regsync would sail through the pre-flight check looking fine. reports_version matches exhaustively with no wildcard, so adding a tool forces that decision rather than defaulting to lenient.

Version parsing moves into a pure function so both paths are testable without spawning a process. The two guards were kill-tested rather than assumed: removing the exit-status check fails failed_probe_is_not_recorded_as_a_version and nothing else, and making every tool exempt fails failed_probe_errors_for_a_tool_that_reports_a_version and nothing else. The dregsy fixture is the exact stderr string the archive actually stored, not an invented one.

check_tool was the only site in xtask ignoring exit status -- report.rs uses .filter(|o| o.status.success()) and remote.rs checks explicitly -- so this makes it consistent with the surrounding code rather than introducing a new pattern.

Two related gaps are left alone, both pre-existing and neither caused here. check_tool spawns the tool by name from PATH while run_tool runs ocync from target/release/ocync, and it runs before build_ocync, so the recorded ocync version is not necessarily the binary that was benchmarked. Separately, skopeo performs every transfer dregsy is credited with but is not a Tool variant, so its version reaches no record. Both need the bench flow reordered or the record schema extended, which is a larger change than this one.

check_tool never inspected the exit status and fell back to stderr, so a
tool that exits non-zero from its version probe had the error text stored
as its version. dregsy has no version flag, so 10 of its 13 records in
bench/results/ecr.json hold a Go flag error where a version belongs, which
defeats the archive's purpose of attributing a metric shift to a version.

Tolerating a failed probe is specific to the tool that has no version to
report. check_tool is the only gate that runs before the benchmark starts,
so for a tool that does report one a failed probe is an error rather than
an absent version. Version parsing moves into a pure function so both paths
are testable without spawning a process.
@bryantbiggs
bryantbiggs merged commit 37175a7 into main Aug 13, 2026
16 checks passed
@bryantbiggs
bryantbiggs deleted the fix/bench-version-probe branch August 13, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant