Attribute skipped tests per file, and re-lower the dead-key ceiling (#132) - #135
Merged
juemerson-at-purestorage merged 3 commits intoAug 21, 2026
Conversation
PR dmann000#134 fixed dmann000#119, removing the two `source.name` records for Invoke-PfbNetworkPing and Invoke-PfbNetworkTrace -- one of them a severity WRONG-RESULTS entry. The committed dead-key report went 85 -> 83. Nothing red, because 85 is a ceiling and a drop is the direction it calls better. That is exactly why it needs re-lowering by hand: left at 85 the gate silently tolerates two brand-new dead keys, and reports safety it is no longer providing. It stays a ceiling rather than becoming a pin. Dead keys legitimately fall as fixes land, and a pin would red every such fix -- making the gate a tax on doing the right thing. The cost of a ceiling is precisely the slack being closed here, so closing it promptly is the discipline the ceiling depends on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n000#132) Replaces the per-edition `MaxSkipped` ceiling in Tests/coverage-baseline.psd1 with `ExpectedSkips`, an exact expected skipped-test count per test file. The ceiling carried deliberate headroom, and the headroom was the defect rather than the mitigation. It did not prevent false reds; it converted an ATTRIBUTABLE red on the PR that moved the number into an UNATTRIBUTABLE red on a later, innocent PR -- and it hid a real coverage regression of up to (ceiling - measured) tests, which is the issue-dmann000#63 failure shape merely bounded in size. That is not hypothetical and the arithmetic was exact: a 5.1 run measured 292 against a ceiling of 268, and six of the +40 belonged to dmann000#120 two days earlier, which had slack and passed without touching the baseline. The next branch across the line was billed for all forty. The five raise notes left in the winps51 block are retained verbatim as the audit trail -- by the last one they are visibly diagnosing the mechanism rather than the branches. Attribution reads $Result.Containers, NOT $Result.Tests, and the difference is load-bearing: a file whose `#requires` stops it before discovery contributes no entries to .Tests at all but still appears in .Containers with every count at zero. Probed directly on both editions (Pester 6.0.1 under pwsh 7, 6.0.0 under Windows PowerShell 5.1) rather than assumed. Three rails come with the map: - no test file may run EMPTY, contributing neither an executed nor a skipped test. This is the dmann000#63 shape below the granularity RequiredDescribes can reach, which only covers hand-listed blocks. - no UNDECLARED file may skip. - a declared file that stops running at all is a violation, not a stale entry to delete -- the tempting reading is the one that turns a regression into a tidy-up. Plus a reconciliation check that per-file skips sum to the run's own total, so a container the walk misses is a red rather than a quietly smaller number. Numbers seeded from run 32392093324 on main@98c7a16, whose Detailed output attributes every skip to a file: 2 across 2 files on all three pwsh legs (which agreed exactly, so one shared pwsh7 map is correct rather than a windows-only measurement generalised), and 297 across 19 files on 5.1. Both sums reconcile with that run's own totals. Only files that skip appear, so this is 21 lines rather than a 199-file census. Verified before pushing: the real baseline passes when fed a stand-in built from those exact per-file numbers, and reds on each of three mutations -- one declared file moving by +1, an undeclared file starting to skip, and a file running empty. Tests/CiCoverageGate.Tests.ps1 grows from 7 to 15 gate tests, including the direction a ceiling structurally could not see: a declared file skipping FEWER tests than expected. Also asserts MaxSkipped is absent from both blocks, since a leftover key nothing reads is a no-op masquerading as a gate. No wire impact: the diff leaves Public/, Private/, the manifest and the root module entirely untouched, so the usual live-FlashBlade verification does not apply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hardcoded 'C:\repo\Tests' root passed all four Windows-side checks and red the ubuntu and macos legs, in two ways at once: - Join-Path throws DriveNotFoundException for a drive letter that does not exist, which is what actually failed all 13 tests. - Even as a bare literal it would have been wrong, because Split-Path -Leaf does not treat '\' as a separator off Windows -- so attribution would key on the whole string and every leaf lookup would silently miss. The gate script itself was never affected: it reads Split-Path -Leaf off a real container's FullName, which carries native separators. Only the test fixture invented a path. The coverage gate itself reported "passed for pwsh7" on the ubuntu leg with both declared files matching, so the 19/2-entry maps are right. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two test-only gate fixes. Both are cases where a gate had gone slack and was reporting protection it no longer provided.
1. Dead-key ceiling: 85 → 83
PR #134 fixed #119, which removed the two
source.namerecords forInvoke-PfbNetworkPingandInvoke-PfbNetworkTrace— one of them aseverity: WRONG-RESULTSentry. The committed report went 85 → 83.Nothing red, because 85 is a ceiling and a drop is the direction it calls better. That is exactly why it needs re-lowering by hand: left at 85, the gate silently tolerates two brand-new dead keys.
It stays a ceiling, not a pin. Dead keys legitimately fall as fixes land, and a pin would red every such fix — making the gate a tax on doing the right thing.
2. Per-file skip attribution, replacing the global ceiling (#132)
Tests/coverage-baseline.psd1's per-editionMaxSkippedbecomesExpectedSkips: an exact expected skipped-test count per test file.The ceiling carried deliberate headroom, and the headroom was the defect rather than the mitigation. It did not prevent false reds; it converted an attributable red on the PR that moved the number into an unattributable red on a later, innocent PR — and it hid a real coverage regression of up to
ceiling - measuredtests, which is the #63 failure shape merely bounded in size.The arithmetic behind that is exact rather than hypothetical: a 5.1 run measured 292 against a ceiling of 268, and six of the +40 belonged to #120 two days earlier, which had slack and passed without touching the baseline. The next branch across the line was billed for all forty. The five raise notes in the
winps51block are retained verbatim as the audit trail — by the last one they are visibly diagnosing the mechanism rather than the branches ("268 was already stale against main before either branch existed").The implementation detail that decides this works
Attribution reads
$Result.Containers, not$Result.Tests, and the difference is load-bearing: a file whose#requiresstops it before discovery contributes no entries to.Testsat all, but still appears in.Containerswith every count at zero. So.Containersis the only place that failure is visible. Probed directly on both editions (Pester 6.0.1 under pwsh 7, 6.0.0 under Windows PowerShell 5.1) rather than assumed.Rails that come with the map
RequiredDescribesreaches, which only covers hand-listed blocks.Where the numbers came from
Seeded from run
32392093324onmain@98c7a16, whose Detailed output attributes every skip to a file — so no local aggregate-suite run and no deliberately-red first push was needed to measure it:Both sums reconcile with that run's own totals, which is the control proving the parse. The three pwsh legs agreeing is what makes one shared
pwsh7map correct rather than a Windows-only measurement generalised. Only files that actually skip appear, so the map is 21 lines rather than a 199-file census.Verification
Container ok, on the three touched test files.Tests/CiCoverageGate.Tests.ps1grows from 7 to 15 gate tests, including the direction a ceiling structurally could not see: a declared file skipping fewer tests than expected. Also assertsMaxSkippedis absent from both blocks, since a leftover key nothing reads is a no-op masquerading as a gate.Live-FlashBlade verification does not apply. The diff is
Tests/andscripts/only, leavingPublic/,Private/,PureStorageFlashBladePowerShell.psd1andPureStorageFlashBladePowerShell.psm1entirely untouched, so it cannot change what goes on the wire.No version bump and no CHANGELOG entry, per the usual convention.
🤖 Generated with Claude Code