Skip to content

Commit 5f90247

Browse files
authored
Collect coverage in the race shards instead of a separate test run (#4376)
Coverage ran every test a second time, in a separate job alongside the race shards. On PRs it used a diff-driven fast path: changed packages plus one level of reverse dependencies, uploaded under a separate partial flag. On pushes it passed `-coverpkg` a comma-separated list of every package, and go test repeats that list on every coverage line, so each line was about 52 KB. That buried failures such as the clist panic in https://github.com/sei-protocol/sei-chain/actions/runs/34233688470/job/102085875556. Sei-DB split its state_db coverage into the same fast and full paths. Both Coverage jobs are removed. The race shards pass `-coverprofile=coverage.out` and each uploads its profile to Codecov under the `sei-chain` flag. The race job in sei-db-tests.yml does the same under `sei-db-state-db`. Merge-queue runs skip the upload. Coverage lines now read "coverage: X% of statements". There is no `-coverpkg` and no state_db filtering, and codecov.yml drops the `sei-chain-pr` flag. Coverage is now measured per package, so a package counts as covered only by its own tests. On this PR the `sei-chain` flag reads 53.6%, against 66.6% on main, which is still above the 40% target. On PRs, Go Test wall-clock is roughly unchanged: 436s against 398s for a main-config run on the same day, with most of the gap down to runner load. Sei-DB is unchanged at 256s. Pushes to main no longer run the separate full coverage pass, which took about 17 minutes.
1 parent 714ecac commit 5f90247

3 files changed

Lines changed: 25 additions & 302 deletions

File tree

‎.github/workflows/go-test.yml‎

Lines changed: 12 additions & 151 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,18 @@ jobs:
6666
| awk -v n="${{ strategy.job-total }}" -v s="${{ matrix.shard }}" '(NR - 1) % n == s' > packages.txt
6767
[[ -s packages.txt ]] || { echo "shard ${{ matrix.shard }} received no packages"; exit 1; }
6868
echo "Shard ${{ matrix.shard }}/${{ strategy.job-total }}: $(wc -l < packages.txt) packages"
69-
xargs -r go test -timeout=${{ env.GO_TEST_TIMEOUT }} < packages.txt
69+
xargs -r go test -timeout=${{ env.GO_TEST_TIMEOUT }} -coverprofile=coverage.out < packages.txt
70+
71+
- name: Upload coverage to Codecov
72+
if: github.event_name != 'merge_group'
73+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
74+
with:
75+
token: ${{ secrets.CODECOV_TOKEN }}
76+
fail_ci_if_error: ${{ github.event_name != 'pull_request' }}
77+
disable_search: 'true'
78+
name: sei-chain-coverage-${{ matrix.shard }}
79+
files: ./coverage.out
80+
flags: sei-chain
7081

7182
upgrade-test:
7283
name: Upgrade tests
@@ -118,153 +129,3 @@ jobs:
118129
exit 1
119130
fi
120131
echo "All race shards and upgrade tests passed."
121-
122-
coverage:
123-
name: Coverage
124-
runs-on: ${{ github.event_name == 'merge_group' && 'ubuntu-latest' || 'uci-default' }}
125-
# Skip coverage report for merge groups, since the queue is about safety check. The merge to main will run coverage anyway.
126-
# GitHub does not support setting "Required" CI workflows for merge queue separately from PRs. If we skip the job at top
127-
# level we then have to work around result not being present. Hence, the repeated if statements per step.
128-
env:
129-
GOFLAGS: -tags=ledger,test_ledger_mock
130-
steps:
131-
- name: Check trigger
132-
if: github.event_name == 'merge_group'
133-
run: echo 'Coverage skipped in merge queue'
134-
135-
# See: https://github.com/actions/checkout/releases/tag/v7.0.0
136-
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0
137-
if: github.event_name != 'merge_group'
138-
with:
139-
# Depth 0 for PRs so merge-base diff against the base branch works.
140-
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
141-
142-
- uses: actions/setup-go@v7
143-
if: github.event_name != 'merge_group'
144-
with:
145-
go-version: ${{ env.GO_VERSION }}
146-
cache: false
147-
148-
- name: Login to Docker Hub
149-
# See: https://github.com/docker/login-action/releases/tag/v4.2.0
150-
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee
151-
if: github.event_name != 'merge_group' && env.DOCKERHUB_USERNAME != ''
152-
env:
153-
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
154-
with:
155-
username: ${{ secrets.DOCKERHUB_USERNAME }}
156-
password: ${{ secrets.DOCKERHUB_TOKEN }}
157-
158-
- name: Download modules
159-
if: github.event_name != 'merge_group'
160-
run: go mod download
161-
162-
- name: Determine coverage packages (PR)
163-
id: cov-pkgs
164-
if: github.event_name == 'pull_request'
165-
run: |
166-
set -euo pipefail
167-
168-
BASE_SHA="${{ github.event.pull_request.base.sha }}"
169-
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
170-
171-
# Use explicit SHAs — works regardless of checkout depth.
172-
CHANGED=$(git diff --name-only "$BASE_SHA"..."$HEAD_SHA")
173-
174-
# Keep only existing .go files (filters out deletions).
175-
CHANGED_GO=$(printf "%s\n" "$CHANGED" \
176-
| grep -E '\.go$' \
177-
| while read -r f; do [ -f "$f" ] && echo "$f"; done \
178-
|| true)
179-
180-
if [[ -z "$CHANGED_GO" ]]; then
181-
echo "skip=true" >> "$GITHUB_OUTPUT"
182-
exit 0
183-
fi
184-
185-
# Resolve changed directories to Go packages, then drop the
186-
# state_db subtree (covered by sei-db-tests.yml) so it neither
187-
# gets tested nor measured for coverage here.
188-
DIRECT_PKGS=$(printf "%s\n" "$CHANGED_GO" \
189-
| xargs -r -n1 dirname \
190-
| sort -u \
191-
| while read -r d; do go list "./$d" 2>/dev/null || true; done \
192-
| sort -u \
193-
| grep -v "^${STATE_DB_PKG_PREFIX}" || true)
194-
195-
if [[ -z "$DIRECT_PKGS" ]]; then
196-
echo "skip=true" >> "$GITHUB_OUTPUT"
197-
exit 0
198-
fi
199-
200-
# Find one level of reverse dependencies (packages that import
201-
# the changed packages) so their tests exercise our changes too.
202-
# Also strip state_db reverse deps; sei-db-tests.yml owns them.
203-
ALL_IMPORTS=$(go list -f '{{.ImportPath}} {{join .Imports " "}} {{join .TestImports " "}}' ./...)
204-
205-
REV_DEPS=$(awk '
206-
NR == FNR { if (NF) targets[$1] = 1; next }
207-
{
208-
pkg = $1
209-
for (i = 2; i <= NF; i++) {
210-
if ($i in targets) { print pkg; break }
211-
}
212-
}
213-
' <(printf "%s\n" "$DIRECT_PKGS") <(printf "%s\n" "$ALL_IMPORTS") \
214-
| sort -u \
215-
| grep -v "^${STATE_DB_PKG_PREFIX}" || true)
216-
217-
# Merge direct + reverse-dep packages, deduplicate.
218-
TEST_PKGS=$(printf "%s\n%s\n" "$DIRECT_PKGS" "$REV_DEPS" \
219-
| awk 'NF && !seen[$0]++ { printf "%s ", $0 }')
220-
TEST_PKGS="${TEST_PKGS% }"
221-
222-
# coverpkg stays as the direct changed packages — we want coverage
223-
# measured on the code that actually changed, but tested via a wider
224-
# set of packages (including reverse deps).
225-
COV_PKGS=$(printf "%s\n" "$DIRECT_PKGS" \
226-
| awk 'NF { printf "%s%s", sep, $0; sep = "," }')
227-
228-
echo "test_packages=$TEST_PKGS" >> "$GITHUB_OUTPUT"
229-
echo "coverpkg=$COV_PKGS" >> "$GITHUB_OUTPUT"
230-
echo "skip=false" >> "$GITHUB_OUTPUT"
231-
232-
- name: Go test with coverage (PR fast path)
233-
if: github.event_name == 'pull_request' && steps.cov-pkgs.outputs.skip != 'true'
234-
run: |
235-
go test \
236-
-timeout=${{ env.GO_TEST_TIMEOUT }} \
237-
-covermode=atomic \
238-
-coverprofile=coverage.out \
239-
-coverpkg=${{ steps.cov-pkgs.outputs.coverpkg }} \
240-
${{ steps.cov-pkgs.outputs.test_packages }}
241-
242-
- name: Go test with coverage (full path)
243-
if: github.event_name != 'merge_group' && github.event_name != 'pull_request'
244-
run: |
245-
set -euo pipefail
246-
PKG_LIST=$(go list ./... | grep -v "^${STATE_DB_PKG_PREFIX}")
247-
PKGS=$(echo "$PKG_LIST" | tr '\n' ' ')
248-
COVERPKG=$(echo "$PKG_LIST" | paste -sd, -)
249-
# Pass $PKGS unquoted so go test gets one invocation with N args,
250-
# not N invocations via xargs (which would each clobber coverage.out).
251-
go test \
252-
-timeout=${{ env.GO_TEST_TIMEOUT }} \
253-
-covermode=atomic \
254-
-coverprofile=coverage.out \
255-
-coverpkg="$COVERPKG" \
256-
$PKGS
257-
258-
- name: Upload coverage to Codecov
259-
if: github.event_name != 'merge_group' && (github.event_name != 'pull_request' || steps.cov-pkgs.outputs.skip != 'true')
260-
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
261-
with:
262-
token: ${{ secrets.CODECOV_TOKEN }}
263-
# Hard-fail on main (full path); soft on PRs (partial data).
264-
fail_ci_if_error: ${{ github.event_name != 'pull_request' }}
265-
disable_search: 'true'
266-
# PR fast-path coverage is intentionally partial.
267-
# Upload under a separate flag to avoid apples-to-oranges project comparisons.
268-
name: ${{ github.event_name == 'pull_request' && 'sei-chain-pr-coverage' || 'sei-chain-coverage' }}
269-
files: ./coverage.out
270-
flags: ${{ github.event_name == 'pull_request' && 'sei-chain-pr' || 'sei-chain' }}

‎.github/workflows/sei-db-tests.yml‎

Lines changed: 12 additions & 144 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,20 @@ jobs:
5151
run: |
5252
go test \
5353
-timeout=${{ env.GO_TEST_TIMEOUT }} \
54+
-coverprofile=coverage.out \
5455
./sei-db/state_db/...
5556
57+
- name: Upload coverage to Codecov
58+
if: github.event_name != 'merge_group'
59+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
60+
with:
61+
token: ${{ secrets.CODECOV_TOKEN }}
62+
fail_ci_if_error: ${{ github.event_name != 'pull_request' }}
63+
disable_search: 'true'
64+
name: sei-db-state-db-coverage
65+
files: ./coverage.out
66+
flags: sei-db-state-db
67+
5668
# The run above compiles the benchmarks but never executes them, so a
5769
# benchmark can stop running without failing anything. One iteration each
5870
# is enough to catch that; the timings it prints are not measurements.
@@ -67,147 +79,3 @@ jobs:
6779
-bench . \
6880
-benchtime=1x \
6981
./sei-db/bench/...
70-
71-
coverage:
72-
name: Coverage
73-
runs-on: ${{ github.event_name == 'merge_group' && 'ubuntu-latest' || 'uci-default' }}
74-
# Skip coverage report for merge groups, since the queue is about safety check. The merge to main will run coverage anyway.
75-
# GitHub does not support setting "Required" CI workflows for merge queue separately from PRs. If we skip the job at top
76-
# level we then have to work around result not being present. Hence, the repeated if statements per step.
77-
env:
78-
GOFLAGS: -tags=ledger,test_ledger_mock
79-
steps:
80-
- name: Check trigger
81-
if: github.event_name == 'merge_group'
82-
run: echo 'Coverage skipped in merge queue'
83-
84-
# See: https://github.com/actions/checkout/releases/tag/v7.0.0
85-
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0
86-
if: github.event_name != 'merge_group'
87-
with:
88-
# Depth 0 for PRs so merge-base diff against the base branch works.
89-
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
90-
91-
- uses: actions/setup-go@v7
92-
if: github.event_name != 'merge_group'
93-
with:
94-
go-version: ${{ env.GO_VERSION }}
95-
cache: false
96-
97-
- name: Login to Docker Hub
98-
# See: https://github.com/docker/login-action/releases/tag/v4.2.0
99-
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee
100-
if: github.event_name != 'merge_group' && env.DOCKERHUB_USERNAME != ''
101-
env:
102-
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
103-
with:
104-
username: ${{ secrets.DOCKERHUB_USERNAME }}
105-
password: ${{ secrets.DOCKERHUB_TOKEN }}
106-
107-
- name: Download modules
108-
if: github.event_name != 'merge_group'
109-
run: go mod download
110-
111-
- name: Determine coverage packages (PR)
112-
id: cov-pkgs
113-
if: github.event_name == 'pull_request'
114-
run: |
115-
set -euo pipefail
116-
117-
BASE_SHA="${{ github.event.pull_request.base.sha }}"
118-
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
119-
120-
# Use explicit SHAs — works regardless of checkout depth.
121-
CHANGED=$(git diff --name-only "$BASE_SHA"..."$HEAD_SHA")
122-
123-
# Keep only existing .go files (filters out deletions).
124-
CHANGED_GO=$(printf "%s\n" "$CHANGED" \
125-
| grep -E '\.go$' \
126-
| while read -r f; do [ -f "$f" ] && echo "$f"; done \
127-
|| true)
128-
129-
if [[ -z "$CHANGED_GO" ]]; then
130-
echo "skip=true" >> "$GITHUB_OUTPUT"
131-
exit 0
132-
fi
133-
134-
# Resolve changed directories to Go packages, then keep only
135-
# the state_db subtree — this workflow exclusively owns those
136-
# packages, and the main go-test.yml workflow owns the rest.
137-
DIRECT_PKGS=$(printf "%s\n" "$CHANGED_GO" \
138-
| xargs -r -n1 dirname \
139-
| sort -u \
140-
| while read -r d; do go list "./$d" 2>/dev/null || true; done \
141-
| sort -u \
142-
| grep "^${STATE_DB_PKG_PREFIX}" || true)
143-
144-
if [[ -z "$DIRECT_PKGS" ]]; then
145-
echo "skip=true" >> "$GITHUB_OUTPUT"
146-
exit 0
147-
fi
148-
149-
# Find one level of reverse dependencies (packages that import
150-
# the changed packages) so their tests exercise our changes too.
151-
# Restrict reverse deps to the state_db subtree as well.
152-
ALL_IMPORTS=$(go list -f '{{.ImportPath}} {{join .Imports " "}} {{join .TestImports " "}}' ./...)
153-
154-
REV_DEPS=$(awk '
155-
NR == FNR { if (NF) targets[$1] = 1; next }
156-
{
157-
pkg = $1
158-
for (i = 2; i <= NF; i++) {
159-
if ($i in targets) { print pkg; break }
160-
}
161-
}
162-
' <(printf "%s\n" "$DIRECT_PKGS") <(printf "%s\n" "$ALL_IMPORTS") \
163-
| sort -u \
164-
| grep "^${STATE_DB_PKG_PREFIX}" || true)
165-
166-
# Merge direct + reverse-dep packages, deduplicate.
167-
TEST_PKGS=$(printf "%s\n%s\n" "$DIRECT_PKGS" "$REV_DEPS" \
168-
| awk 'NF && !seen[$0]++ { printf "%s ", $0 }')
169-
TEST_PKGS="${TEST_PKGS% }"
170-
171-
# coverpkg stays as the direct changed packages — we want coverage
172-
# measured on the code that actually changed, but tested via a wider
173-
# set of packages (including reverse deps).
174-
COV_PKGS=$(printf "%s\n" "$DIRECT_PKGS" \
175-
| awk 'NF { printf "%s%s", sep, $0; sep = "," }')
176-
177-
echo "test_packages=$TEST_PKGS" >> "$GITHUB_OUTPUT"
178-
echo "coverpkg=$COV_PKGS" >> "$GITHUB_OUTPUT"
179-
echo "skip=false" >> "$GITHUB_OUTPUT"
180-
181-
- name: Go test with coverage (PR fast path)
182-
if: github.event_name == 'pull_request' && steps.cov-pkgs.outputs.skip != 'true'
183-
run: |
184-
go test \
185-
-timeout=${{ env.GO_TEST_TIMEOUT }} \
186-
-covermode=atomic \
187-
-coverprofile=coverage.out \
188-
-coverpkg=${{ steps.cov-pkgs.outputs.coverpkg }} \
189-
${{ steps.cov-pkgs.outputs.test_packages }}
190-
191-
- name: Go test with coverage (full path)
192-
if: github.event_name != 'merge_group' && github.event_name != 'pull_request'
193-
run: |
194-
go test \
195-
-timeout=${{ env.GO_TEST_TIMEOUT }} \
196-
-covermode=atomic \
197-
-coverprofile=coverage.out \
198-
-coverpkg=./sei-db/state_db/... \
199-
./sei-db/state_db/...
200-
201-
- name: Upload coverage to Codecov
202-
if: github.event_name != 'merge_group' && (github.event_name != 'pull_request' || steps.cov-pkgs.outputs.skip != 'true')
203-
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
204-
with:
205-
token: ${{ secrets.CODECOV_TOKEN }}
206-
# Hard-fail on main (full path); soft on PRs (partial data).
207-
fail_ci_if_error: ${{ github.event_name != 'pull_request' }}
208-
disable_search: 'true'
209-
# PR fast-path coverage is intentionally partial.
210-
# Upload under a separate flag to avoid apples-to-oranges project comparisons.
211-
name: ${{ github.event_name == 'pull_request' && 'sei-db-state-db-pr-coverage' || 'sei-db-state-db-coverage' }}
212-
files: ./coverage.out
213-
flags: ${{ github.event_name == 'pull_request' && 'sei-db-state-db-pr' || 'sei-db-state-db' }}

‎codecov.yml‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,13 @@ coverage:
44
status:
55
project:
66
default:
7-
informational: true # PR fast-path uploads are partial; avoid scary global regressions.
7+
informational: true
88
target: auto
99
threshold: 3% # allow this much decrease on project
1010
sei-chain:
1111
target: 40%
1212
flags:
1313
- sei-chain
14-
sei-chain-pr:
15-
informational: true
16-
flags:
17-
- sei-chain-pr
1814
patch:
1915
default:
2016
informational: true # Show patch coverage, but do not fail CI for it.
@@ -58,5 +54,3 @@ ignore:
5854
flags:
5955
sei-chain:
6056
carryforward: true
61-
sei-chain-pr:
62-
carryforward: false

0 commit comments

Comments
 (0)