Skip to content

Attribute skipped tests per file, and re-lower the dead-key ceiling (#132) - #135

Merged
juemerson-at-purestorage merged 3 commits into
dmann000:mainfrom
juemerson-at-purestorage:fix/stale-coverage-gates
Aug 21, 2026
Merged

juemerson-at-purestorage merged 3 commits into
dmann000:mainfrom
juemerson-at-purestorage:fix/stale-coverage-gates

Conversation

@juemerson-at-purestorage

Copy link
Copy Markdown
Collaborator

Two test-only gate fixes. Both are cases where a gate had gone slack and was reporting protection it no longer provided.

1. Dead-key ceiling: 85 → 83

PR #134 fixed #119, which removed the two source.name records for Invoke-PfbNetworkPing and Invoke-PfbNetworkTrace — one of them a severity: WRONG-RESULTS entry. The committed report went 85 → 83.

Nothing red, because 85 is a ceiling and a drop is the direction it calls better. That is exactly why it needs re-lowering by hand: left at 85, the gate silently tolerates two brand-new dead keys.

It stays a ceiling, not a pin. Dead keys legitimately fall as fixes land, and a pin would red every such fix — making the gate a tax on doing the right thing.

2. Per-file skip attribution, replacing the global ceiling (#132)

Tests/coverage-baseline.psd1's per-edition MaxSkipped becomes ExpectedSkips: an exact expected skipped-test count per test file.

The ceiling carried deliberate headroom, and the headroom was the defect rather than the mitigation. It did not prevent false reds; it converted an attributable red on the PR that moved the number into an unattributable red on a later, innocent PR — and it hid a real coverage regression of up to ceiling - measured tests, which is the #63 failure shape merely bounded in size.

The arithmetic behind that is exact rather than hypothetical: a 5.1 run measured 292 against a ceiling of 268, and six of the +40 belonged to #120 two days earlier, which had slack and passed without touching the baseline. The next branch across the line was billed for all forty. The five raise notes in the winps51 block are retained verbatim as the audit trail — by the last one they are visibly diagnosing the mechanism rather than the branches ("268 was already stale against main before either branch existed").

The implementation detail that decides this works

Attribution reads $Result.Containers, not $Result.Tests, and the difference is load-bearing: a file whose #requires stops it before discovery contributes no entries to .Tests at all, but still appears in .Containers with every count at zero. So .Containers is the only place that failure is visible. Probed directly on both editions (Pester 6.0.1 under pwsh 7, 6.0.0 under Windows PowerShell 5.1) rather than assumed.

Rails that come with the map

  • No test file may run empty — contributing neither an executed nor a skipped test. This is the CI silently skips ~23% of the test suite, including the absolute-path regression guards #63 shape below the granularity RequiredDescribes reaches, which only covers hand-listed blocks.
  • No undeclared file may skip.
  • A declared file that stops running at all is a violation, not a stale entry to delete. The tempting reading is the one that turns a regression into a tidy-up.
  • Per-file skips must reconcile with the run's own total, so a container the walk misses is a red rather than a quietly smaller number.

Where the numbers came from

Seeded from run 32392093324 on main@98c7a16, whose Detailed output attributes every skip to a file — so no local aggregate-suite run and no deliberately-red first push was needed to measure it:

Leg Skips Files
pwsh 7 (ubuntu, windows, macos — all three agreed exactly) 2 2
Windows PowerShell 5.1 297 19

Both sums reconcile with that run's own totals, which is the control proving the parse. The three pwsh legs agreeing is what makes one shared pwsh7 map correct rather than a Windows-only measurement generalised. Only files that actually skip appear, so the map is 21 lines rather than a 199-file census.

Verification

  • Scoped Pester, both editions: 32 passed / 0 failed / 0 skipped, Container ok, on the three touched test files.
  • Tests/CiCoverageGate.Tests.ps1 grows from 7 to 15 gate tests, including the direction a ceiling structurally could not see: a declared file skipping fewer tests than expected. Also asserts MaxSkipped is absent from both blocks, since a leftover key nothing reads is a no-op masquerading as a gate.
  • End-to-end against the real baseline: it passes when fed a stand-in built from the exact per-file numbers above, and reds on each of three mutations — one declared file moving by +1, an undeclared file starting to skip, and a file running empty.

Live-FlashBlade verification does not apply. The diff is Tests/ and scripts/ only, leaving Public/, Private/, PureStorageFlashBladePowerShell.psd1 and PureStorageFlashBladePowerShell.psm1 entirely untouched, so it cannot change what goes on the wire.

No version bump and no CHANGELOG entry, per the usual convention.

🤖 Generated with Claude Code

PR dmann000#134 fixed dmann000#119, removing the two `source.name` records for
Invoke-PfbNetworkPing and Invoke-PfbNetworkTrace -- one of them a
severity WRONG-RESULTS entry. The committed dead-key report went
85 -> 83.

Nothing red, because 85 is a ceiling and a drop is the direction it
calls better. That is exactly why it needs re-lowering by hand: left
at 85 the gate silently tolerates two brand-new dead keys, and reports
safety it is no longer providing.

It stays a ceiling rather than becoming a pin. Dead keys legitimately
fall as fixes land, and a pin would red every such fix -- making the
gate a tax on doing the right thing. The cost of a ceiling is precisely
the slack being closed here, so closing it promptly is the discipline
the ceiling depends on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n000#132)

Replaces the per-edition `MaxSkipped` ceiling in Tests/coverage-baseline.psd1
with `ExpectedSkips`, an exact expected skipped-test count per test file.

The ceiling carried deliberate headroom, and the headroom was the defect
rather than the mitigation. It did not prevent false reds; it converted an
ATTRIBUTABLE red on the PR that moved the number into an UNATTRIBUTABLE red
on a later, innocent PR -- and it hid a real coverage regression of up to
(ceiling - measured) tests, which is the issue-dmann000#63 failure shape merely
bounded in size.

That is not hypothetical and the arithmetic was exact: a 5.1 run measured 292
against a ceiling of 268, and six of the +40 belonged to dmann000#120 two days
earlier, which had slack and passed without touching the baseline. The next
branch across the line was billed for all forty. The five raise notes left in
the winps51 block are retained verbatim as the audit trail -- by the last one
they are visibly diagnosing the mechanism rather than the branches.

Attribution reads $Result.Containers, NOT $Result.Tests, and the difference
is load-bearing: a file whose `#requires` stops it before discovery
contributes no entries to .Tests at all but still appears in .Containers with
every count at zero. Probed directly on both editions (Pester 6.0.1 under
pwsh 7, 6.0.0 under Windows PowerShell 5.1) rather than assumed.

Three rails come with the map:

  - no test file may run EMPTY, contributing neither an executed nor a
    skipped test. This is the dmann000#63 shape below the granularity
    RequiredDescribes can reach, which only covers hand-listed blocks.
  - no UNDECLARED file may skip.
  - a declared file that stops running at all is a violation, not a stale
    entry to delete -- the tempting reading is the one that turns a
    regression into a tidy-up.

Plus a reconciliation check that per-file skips sum to the run's own total,
so a container the walk misses is a red rather than a quietly smaller number.

Numbers seeded from run 32392093324 on main@98c7a16, whose Detailed output
attributes every skip to a file: 2 across 2 files on all three pwsh legs
(which agreed exactly, so one shared pwsh7 map is correct rather than a
windows-only measurement generalised), and 297 across 19 files on 5.1. Both
sums reconcile with that run's own totals. Only files that skip appear, so
this is 21 lines rather than a 199-file census.

Verified before pushing: the real baseline passes when fed a stand-in built
from those exact per-file numbers, and reds on each of three mutations -- one
declared file moving by +1, an undeclared file starting to skip, and a file
running empty.

Tests/CiCoverageGate.Tests.ps1 grows from 7 to 15 gate tests, including the
direction a ceiling structurally could not see: a declared file skipping
FEWER tests than expected. Also asserts MaxSkipped is absent from both
blocks, since a leftover key nothing reads is a no-op masquerading as a gate.

No wire impact: the diff leaves Public/, Private/, the manifest and the root
module entirely untouched, so the usual live-FlashBlade verification does not
apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hardcoded 'C:\repo\Tests' root passed all four Windows-side checks and red
the ubuntu and macos legs, in two ways at once:

- Join-Path throws DriveNotFoundException for a drive letter that does not
  exist, which is what actually failed all 13 tests.
- Even as a bare literal it would have been wrong, because Split-Path -Leaf
  does not treat '\' as a separator off Windows -- so attribution would key on
  the whole string and every leaf lookup would silently miss.

The gate script itself was never affected: it reads Split-Path -Leaf off a real
container's FullName, which carries native separators. Only the test fixture
invented a path. The coverage gate itself reported "passed for pwsh7" on the
ubuntu leg with both declared files matching, so the 19/2-entry maps are right.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@juemerson-at-purestorage
juemerson-at-purestorage merged commit 7304acb into dmann000:main Aug 21, 2026
5 checks passed
@juemerson-at-purestorage
juemerson-at-purestorage deleted the fix/stale-coverage-gates branch August 25, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Invoke-PfbNetworkPing / Invoke-PfbNetworkTrace: -SourceName sends the wrong query key and always fails with HTTP 400

1 participant