Skip to content

Ask the toolchain which files are ours, in one place - #3

Merged
verygoodsoftwarenotvirus merged 1 commit into
mainfrom
one-source-of-truth-for-the-go-file-list
Aug 15, 2026
Merged

verygoodsoftwarenotvirus merged 1 commit into
mainfrom
one-source-of-truth-for-the-go-file-list

Conversation

@verygoodsoftwarenotvirus

Copy link
Copy Markdown
Contributor

The problem

The tools that walk the filesystem each decided for themselves which Go files belong to this module, and they disagreed three ways:

Where How it excluded vendor
scripts/format_golang.sh find … -not -path '*/vendor/*'
scripts/format_imports.sh find … -not -path '*/vendor/*'
.github/workflows/formatting.yaml gofmt -l . | grep -Ev '^vendor\/'
scripts/goimports.sh it didn't — go tool goimports -w .

That last row is a bug rather than a duplication. goimports -w . walks from the working directory, so with a vendor tree present make format rewrote vendored third-party source. Measured against this repo's own dependencies: 171 of 2681 vendored files reformatted before I interrupted it, and the run was still going after six minutes.

The fix

scripts/go_files.sh answers the question once, by asking the Go toolchain:

go list -e -f '{{.Dir}}' ./...   # then find -maxdepth 1 over those directories

./... does not descend into vendor/, testdata/, or any _- or .-prefixed directory — which is exactly why every wildcard-driven target here (test, lint, go fix, fieldalignment, tagalign) has never needed an exclusion at all. Only the filesystem walkers ever did, and now they inherit the same answer instead of each maintaining their own.

-e keeps a package that does not compile in the list, because formatting a file is most useful exactly when it is still broken. -maxdepth 1 because what go list names are package directories, and a package is entitled to a testdata directory of its own.

Two details worth reviewing closely

Both were found while testing this, and both are the sort of thing that hides:

It fails loudly rather than emitting an empty list. An out-of-sync vendor/modules.txt makes go list exit non-zero — not exotic, it's what you get from editing go.mod without re-vendoring. The first version of this returned an empty list in that case, and every formatter silently formatted nothing while the CI check reported "clean". Silence read as success.

Callers read it through a file, not < <(go_files.sh). Process substitution discards the exit status of what it runs, which is precisely how that empty list went unnoticed. The workflow sets pipefail for the same reason: xargs exits 0 on no input and would otherwise mask the failure.

Verification

Against a real go mod vendor of this module's dependencies (2681 vendored Go files):

Check Before After
Vendored files rewritten by make format 171+ (interrupted) 0
make format wall clock >6 min, unfinished 19s
make format on a clean tree — no-op
CI formatting check — clean
Every path when go list fails silent success exits non-zero

shellcheck passes on all of scripts/, and go test -race ./... is unchanged.

Provenance

This came out of primandproper/tarpaulin, which had drifted its own copies of the same exclusion (*/vendor/* in the scripts, ./vendor/* in the workflow) and has the equivalent change applied. Nothing here is tarpaulin-specific.

The tools that walk the filesystem each decided for themselves which Go
files belong to this module, and they disagreed three ways:
format_golang.sh and format_imports.sh carried their own
`-not -path '*/vendor/*'`, the formatting workflow filtered gofmt's
output with `grep -Ev '^vendor\/'`, and goimports.sh ran
`goimports -w .` with no exclusion at all.

That last one is a bug rather than a duplication. With a vendor tree
present, `make format` rewrote vendored third-party source: on this
repo's own dependencies it had reformatted 171 of 2681 vendored files
before it was interrupted, and the run took over six minutes.

scripts/go_files.sh answers the question once, by asking the Go
toolchain: `go list -e -f '{{.Dir}}' ./...`, then find -maxdepth 1 over
those directories. `./...` does not descend into vendor, testdata, or
any _ or . prefixed directory, which is why every wildcard-driven target
here — test, lint, go fix, fieldalignment, tagalign — has never needed an
exclusion. The walkers now inherit the same answer instead of
maintaining their own.

Two details are load-bearing, both found while testing this:

An out-of-sync vendor/modules.txt makes `go list` exit non-zero. The
first version of this returned an empty list in that case, and every
formatter silently formatted nothing while the CI check reported clean —
silence reading as success. go_files.sh now refuses to emit an empty
list.

Its callers read it through a file rather than `< <(go_files.sh)`,
because process substitution discards the exit status of what it runs,
which is precisely how that empty list went unnoticed. The workflow sets
pipefail for the same reason: xargs exits 0 on no input and would
otherwise mask the failure.

Verified against a real `go mod vendor` of this module's dependencies:
0 of 2681 vendored files touched, `make format` a no-op on a clean tree
and down to 19 seconds, the formatting check clean, and every path
exiting non-zero when the list cannot be produced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TGxR7rXoXGdHXaUxNe346q

@verygoodsoftwarenotvirus verygoodsoftwarenotvirus left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@verygoodsoftwarenotvirus
verygoodsoftwarenotvirus merged commit df464d2 into main Aug 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant