fix(agc): drop an unused stencil test value export from draws without a stencil test

[tv7]

What

Since 22d662e a pixel shader with DB_SHADER_CONTROL.STENCIL_TEST_VAL_EXPORT_ENABLE needs VK_EXT_shader_stencil_export, which NVIDIA's driver does not offer. On NVIDIA this stops Marvel's Wolverine (PPSA03671): sceAgcLinkShaders resolves the ABI of pixel shader 0x11d01d0900 (DB_SHADER_CONTROL bit 1 set), the driver prepares it right there, the recompile throws a pixel shader that exports the stencil reference needs VK_EXT_shader_stencil_export, which the device lacks, and the title aborts.

The exported value is only the stencil test value: it feeds the stencil test and what REPLACE_TEST writes. In a draw without a stencil test it has no effect. So:

• Driver::decodeDraw clears stencilRefExportEnable when the decoded state has no stencil test. The shader then has no FragStencilRefEXT output, on every device. Early fragment tests stay off for it, as before.
• PlanRegistered does not prepare such a pixel shader when its ABI is resolved on a device without the extension, because the draw's stencil test is not known yet there. It is prepared at its first draw instead, like other shaders the ABI step cannot decide.
• Unchanged: a draw that has a stencil test with this shader still throws on a device without the extension, with the same message.

Tested

Windows 11, RTX 3070 (no VK_EXT_shader_stencil_export), GCC 15.2.0 UCRT, Release, on main 0214069e3.

• New agc_driver_stencil_ref_export_unused (GPU): registers a pixel shader that exports a stencil test value and a color, resolves its ABI with DB_SHADER_CONTROL=0x802 and SPI_SHADER_Z_FORMAT=9, then draws it with DB_DEPTH_CONTROL=0 and no depth target, and checks that the color target was written. On main it fails at the ABI resolve with the error above. With only the PlanRegistered change it fails at the draw with the same error. With both it passes. Not run on a device with the extension (none here); there only the draw-time change applies.
• ctest 1096/1096 (the existing agc_driver_stencil_ref_export skips here: the device lacks the extension). tools/check_conventions.py passes.
• Wolverine, on main 682ef0c17 plus feat(agc): implement image and sampler tables indexed at runtime offsets #757 (rebased), fix(agc): render a color view of several array slices into its first slice #3633, fix(agc): store D16 depth with stencil as D24 or D32 where the device lacks it #3846 and feat(agc): decode GS_FAST_LAUNCH geometry with one point per subgroup #4702 (current main does not reach this point without feat(agc): implement image and sampler tables indexed at runtime offsets #757): without this change it aborts with the error above. With it, the title gets past sceAgcLinkShaders and stops later at a separate GetImageResource dword 0 is not a valid runtime value in another pixel shader, before any draw with the stencil-exporting shader. So the draw-time part is covered by the test, not yet by the title.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes, Claude Code

🤖 Generated with Claude Code

[Sam-mon7]

Demon's Souls (PPSA01341 v1.07) stops on the same recompile error on NVIDIA, but in a draw that has the stencil test enabled, so the "no stencil test" rule here doesn't cover it.

Windows 11, RTX 4070 (driver 596.21, no VK_EXT_shader_stencil_export), MinGW GCC 15.2 Release.

• Stop: character creation, after choosing a body type: shader 0x249f10800 stage 5: ... a pixel shader that exports the stencil reference needs VK_EXT_shader_stencil_export, which the device lacks.
• Captured draw registers: DB_SHADER_CONTROL=0x52 (STENCIL_TEST_VAL_EXPORT_ENABLE), DB_DEPTH_CONTROL=0x007007e7 (stencil enabled, STENCILFUNC and STENCILFUNC_BF = ALWAYS, depth test GEQUAL with writes), DB_STENCIL_CONTROL=0 (every op KEEP), DB_STENCILREFMASK/_BF=0x01ffffff, SPI_SHADER_Z_FORMAT=9.
• With ALWAYS on both faces and only KEEP ops, the exported value is never read. Locally I widened the condition from "no stencil test" to "no face consumes the reference". A face consumes it when it compares (compareMask != 0 and the compare op is not ALWAYS or NEVER) or writes it (writeMask != 0 and a fail, pass or depth-fail op is REPLACE). With that change the draw records and the title gets past body-type selection. That tree carries other local changes, so it is not an isolated run of this head.
• Tests for it: the captured register set expects no export. For each face, switching the compare op to EQUAL, or any one op to REPLACE, expects the extension error again. Clearing compareMask or writeMask respectively expects no export again.

Happy for this to be folded in here, or to open it as a follow-up on top of this PR.

AI-assisted: yes

[CnRJay]

The reasoning holds: without a stencil test, the exported test value has no effect.

One pipeline-count side effect to check. The draw-time change (if (!product->state.stencilTest) product->pixel.stencilRefExportEnable = false;) runs on every device, including ones that have VK_EXT_shader_stencil_export. stencilRefExportEnable decides whether the module has a FragStencilRefEXT output, so it is part of what the pixel variant (and the pipeline) is keyed on. A title that uses one stencil-exporting pixel shader both with and without a stencil test then gets two variants and two pipeline sets where main had one, on AMD and Intel, which currently work. It's an extra compile and an extra pipeline on first use for each such shader, so a small hitch, but it's paid on the devices that didn't need the change.

Applying the clear only when the device lacks the extension (the same check PlanRegistered does) keeps one variant on those devices, and gives NVIDIA the fix. If an unconditional clear is preferred because a shader without the export is cheaper (early tests could in principle be allowed), that's a separate decision and would want its own measurement.

On Sam-mon7's Demon's Souls case: widening the rule to "the stencil test can't observe the value" (both faces ALWAYS and every op KEEP, or a zero ref mask) is exact by the same argument and covers that stop too. It needs the full register set in the condition: STENCILFUNC/_BF, the fail/zfail/pass ops of both faces, and REPLACE_TEST in DB_STENCIL_CONTROL.

[fdfawef324]

I have two verified follow-up commits based directly on this PR's current head dbd015c: compare the changes.

The predicate also omits the export when enabled stencil testing cannot consume the reference. It considers both faces, effective 8-bit masks, reachable depth/stencil operations, and preserves the export for dependent comparisons or reachable REPLACE writes. Unknown states remain conservative. The existing clear path and missing-extension refusal are preserved.

Commits available to cherry-pick:

• d699da6 — fix(agc): omit reference-independent stencil exports
• dd7884c — test(agc): check draws with an unused stencil reference

Verification on this public branch: typed CPU checks cover 22 accepted and 12 dependent states, the old predicate fails the ALWAYS/KEEP regression, all four modified C++ units pass optimized syntax checks, and conventions pass against both this PR head and upstream/main. The native fixture extends the existing homebrew programs with full color/stencil readbacks and six dependent-state refusal cases; this public branch has not received a full build or GPU run.

Separate private integration evidence using the literal same production predicate: 20 expected native outcomes passed across the old/new drivers, including KEEP/INVERT, dependent refusals, reference-dependent replay controls, and GPL-disabled positive cases. Demon's Souls moved past the previous pixel-stage refusal to the full character-creation menu with a rendered knight model and no fatal error during a 905.7-second run. The first level is still unverified. Dreaming Sarah remained at 59.94 FPS in the steady-state check.

GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100. GPU/title results above are from the private integration, not a GPU execution of this public branch.

AI used: yes, Codex. Source implementation, independent adversarial review, CPU verification, and root-owned native/title verification completed before publication.

Co-Authored-By: Codex noreply@openai.com

[tv7]

Thanks all three. Rebased onto main e34c479, new head cbf4968:

• @fdfawef324: cherry-picked your two commits as they are (fix(agc): omit reference-independent stencil exports, test(agc): check draws with an unused stencil reference, your authorship kept). That covers @Sam-mon7's Demon's Souls state (both faces ALWAYS, every op KEEP).
• @CnRJay: new commit fix(agc): keep the stencil export on devices with VK_EXT_shader_stencil_export. The draw decode now drops the export only when the device lacks the extension, so AMD and Intel keep one pixel variant per shader as on main. The debt entry in docs/dev/debt/libSceAgcDriver.md is updated to the wider rule.

Windows 11, MinGW GCC 15.2 Release, NVIDIA GeForce RTX 3070 (driver 617.14), no VK_EXT_shader_stencil_export: full build, ctest 1199/1199 (13 skipped: no device for those tests here), conventions check passes. agc_driver_graphics reports 22 accepted states and 12 dependent countermodels, and all eight agc_driver_stencil_ref_export_unused_tests --case runs pass on the GPU (keep and invert draw with full readbacks; the six dependent cases keep the extension refusal).

[adriadam10]

Checked the new head: the unused-reference predicate, the device-gated drop in Driver::decodeDraw (devices with VK_EXT_shader_stencil_export keep one pixel variant), the PlanRegistered deferral and the debt entry. agc_driver_graphics and agc_driver_stencil_ref_export_unused pass; both fail to build without the change.

Added one commit on your branch: removed the code comment in DrawDecode.cpp (no comments except technical debt). Merging once CI is green.

[jotitadev]

Run on a device that has VK_EXT_shader_stencil_export, since the runs so far are on NVIDIA without it. @CnRJay's question was about that case.

GPUs tested: Apple M2 Max, MoltenVK 1.4.2 (Vulkan 1.4.357), macOS 27.2, x86-64 under Rosetta

main does not run the GPU tests on macOS yet, so I tested on macos-port at b7eb6b6b (mugurc/AnyPS5) plus the ImportWatched change from #1, with this PR's own diff (merge-base..head cbf496835) applied. Not main. The only later commit, 30f56882f, removes a code comment in DrawDecode.cpp, so these results also hold for the current head. Release build, LunarG Vulkan SDK 1.4.363.0. Suite: ctest -R "^agc" -E "agc_display_buffer_dcc|agc_driver_flip|agc_driver_lds_atomics64" -j1 --timeout 30. Without the PR, 13 tests fail on this setup (7 with the #2704 build), all known macOS failures listed in #1.

DrawDecode.cpp and tests/Graphics.cpp conflict with code that is on main and not on the branch. I kept the branch's side and added only this PR's own lines:

• the stencil block in decodeDraw;
• stencilComparison and stencilReferenceUnusedTests, with their call and the --unused-stencil-reference switch.

The other files apply cleanly.

• agc_driver_stencil_ref_export_unused passes.
• --case keep and --case invert pass. The six dependent cases are skipped by design (skipped: a dependent reference refusal requires a device without native stencil export).
• agc_driver_graphics passes (22 accepted states, 12 dependent countermodels), and so does agc_driver_stencil_ref_export.
• Suite: the same 13 failures with and without the PR. None new, none fixed.
• With only the new execution test on the unchanged branch, it also passes. That is expected: with the extension present, main never refused this shader. So this run shows that the "keep the export when the extension is present" path does not break anything. It does not exercise the fix itself.

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[adriadam10]

Rebased onto current main (one include conflict in ShaderRegistry.cpp, kept both). agc_driver_graphics, agc_driver_stencil_ref_export and agc_driver_stencil_ref_export_unused pass on the rebased tree. Merging once CI is green.

[tv7]

@adriadam10 thanks for rebasing. I rebased locally at the same time (onto main 9a22507) and only saw yours when my push was refused, so nothing of mine was pushed. My local tree has the same changes as your head ea4378c5a, line for line. On it, on Windows 11 with an RTX 3070: ctest 1222/1223, where the one failure, guest_user_service with 0xc0000409, fails the same way on plain main 9a22507 here. All eight agc_driver_stencil_ref_export_unused_tests --case runs pass, and conventions pass.

[adriadam10]

Rebased onto current main again: the only conflict was the STENCIL_TEST_VAL_EXPORT_ENABLE line in docs/dev/debt/libSceAgcDriver.md, which #5466 changed. Kept main's wording (bounded replay on devices without the extension) and added the StencilReferenceUnused drop on top, in a separate docs commit. agc_driver_graphics, agc_driver_stencil_ref_export and agc_driver_stencil_ref_export_unused pass on the rebased tree. Merging once CI is green.

[fdfawef324]

Demon's Souls (PPSA01342) reaches this refusal on main. Windows Server 2025, RTX 3060 12 GB (no VK_EXT_shader_stencil_export), driver 591.86.

Build: main 2559e9623 + #5870 (boot abort) + #5828 (TYPE 0 T#s) + a follow-up of mine for static V#s with a nonzero TYPE (not opened yet, depends on #5828). That gets the title past its earlier stops into loading the first level from the debug start menu. 1,672 s in, it stops on:

what():  shader 0x24ade3700 stage 5: ShaderRecompiler::Recompile: stencil reference export emulation requires a replay-safe pixel shader with late tests

Our integration builds carry this PR's change (the StencilReferenceUnused drop). On those, the title gets through the same level load and plays in the level, in runs over the last day of up to about 16 minutes in the level. None of them hit this refusal. An earlier capture of the same kind of draw showed stencil ALWAYS/KEEP on both faces with depth GEQUAL, which is the state this PR drops the export for. I did not decode this run's draw state, and I haven't run main + this PR alone.

AI-assisted: yes

[fdfawef324]

Prepared an author-preserving rebase onto main 1db84d574, tip 618b80067c43e43eb6b185168f14d9fcc94fabab.

The conflict is next to the new quad coverage assignments in DrawDecode.cpp. The resolution retains both QuadPixelMask/QuadSampleMasks assignments and this PR's unchanged missing-extension stencil conditional. All six original author records, author dates and complete commit messages are preserved. A separately constructed expected merge tree matches exactly.

Fresh CPU validation against the rebased State.hpp: the literal production predicate and unchanged public sample oracle pass 22 accepted states and 12 dependent counterexamples; the older disable-only predicate fails the intended ALWAYS/KEEP case. Conventions pass against the exact new base. This is source/CPU verification, without a full driver link or native/GPU run on the rebased branch.

One retained test caveat: the six manual dependent --case modes still require the old missing-extension error text. With #5466 on main, a dependent export may use replay or produce the newer replay-safety refusal. Those legacy assertions need validation/update; this rebase does not change their expectations or claim they pass.

GPUs tested: none for this rebase (CPU only). Host: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100.

AI-assisted: yes

Co-Authored-By: Codex noreply@openai.com

[fdfawef324]

Additional native verification: the default unused-export test passes full color/stencil readbacks on the RTX 3060. The replay suite also passes with and without graphics pipeline libraries. All three returned exit 0, in 1.17–1.28 seconds per process.

These ran in a fresh Release/O3 integration containing this PR's unchanged six commits, the earlier deferred-descriptor integration, #6259, and opt-in ownership diagnostics. This is separate from the current-main rebase branch offered above; it does not certify that branch's full driver build, the six legacy dependent --case modes, or title performance.

GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100.

AI-assisted: yes

Co-Authored-By: Codex noreply@openai.com

[adriadam10]

Rebased onto current main again: the only conflict was in DrawDecode.cpp, next to the new quadPixelMask/quadSampleMask assignments (kept both). The previous Windows run failed on agc_interpolant_mapping, which printed "passed" and then hit the 120 s timeout on exit; this diff doesn't touch it. agc_driver_graphics, agc_driver_stencil_ref_export and agc_driver_stencil_ref_export_unused pass on the rebased tree. Merging once CI is green.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#6235	core/libs/prx/libSceAgcDriver/tests/Graphics.cpp L6755