fix(agc): export the stencil reference from pixel shaders with VK_EXT_shader_stencil_export

[oneandonlydean]

What

Refs #4276. Demon's Souls (PPSA01341) stops at depth export, shader coverage or ordered fragment execution is unsupported (register 0x203 = 0x00000052): DB_SHADER_CONTROL.STENCIL_TEST_VAL_EXPORT_ENABLE (bit 1), a pixel shader that writes the stencil test value. Nothing emitted FragStencilRefEXT or enabled VK_EXT_shader_stencil_export.

This does not get the title past the stop on an NVIDIA GPU. NVIDIA's Vulkan driver does not offer VK_EXT_shader_stencil_export (checked on 610.57.04, RTX 4090), and the issue was reported on an RTX 5070 Ti. There the draw now stops with a pixel shader that exports the stencil reference needs VK_EXT_shader_stencil_export, which the device lacks instead of the register message. The draw runs on devices that have the extension (lavapipe has it). A path for devices without it is a separate piece of work and needs the draw's stencil state first, see the question at the end.

How the shader exports the value, from Mesa at 3e32b0e1 (the drivers for this GPU family; nothing here is measured on a console):

• The value leaves in the MRTZ export, target 8 (emit_ps_mrtz_export). With SPI_SHADER_Z_FORMAT 32_GR or 32_ABGR it is the raw integer in the second channel, enable bit 1 (L230-L235); depth is the first channel, the sample mask the third. With UINT16_ABGR it is bits 16 to 23 of the first channel of a compressed export (L209-L218). Which format goes with which exports: ac_get_spi_shader_z_format (a stencil export alone uses 32_GR).
• RADV sets STENCIL_TEST_VAL_EXPORT_ENABLE from "the shader writes FragStencilRef" (radv_shader.c L2037-L2041), next to Z_EXPORT_ENABLE and MASK_EXPORT_ENABLE, and leaves Z_ORDER and KILL_ENABLE as for any other shader (L2021-L2035). So the issue's 0x52 is what RADV programs for a shader that discards and writes the stencil reference.
• The Vulkan reference is STENCILTESTVAL of both DB_STENCILREFMASK registers (radv_cmd_buffer.c L13359-L13360), and VK_STENCIL_OP_REPLACE is STENCIL_REPLACE_TEST (radv_pipeline_graphics.h L404-L405). So the exported value takes the place of the test value: what the comparison uses on both faces and what REPLACE_TEST writes. This is what FragStencilRefEXT does on the host.
• Fields: gfx10.json DB_SHADER_CONTROL.

Inferred, not stated by the sources: that the stencil operations STENCIL_ONES and STENCIL_REPLACE_OP keep writing all ones and STENCILOPVAL when the test value is exported. Mesa never sets STENCIL_OP_VAL_EXPORT_ENABLE (bit 2) and always programs an operation value of 1, so the sources say nothing about bit 2 either.

The change:

• DB_SHADER_CONTROL bit 1 is accepted when SPI_SHADER_Z_FORMAT is 32_GR or 32_ABGR. UINT16_ABGR and the formats without a second channel are refused as before, and so is bit 2.
• The recompiler declares FragStencilRefEXT (capability StencilExportEXT, SPV_EXT_shader_stencil_export, execution mode StencilRefReplacingEXT) for an MRTZ export with enable bit 1 when the pixel stage has the bit set, and stores the second channel into it. The host uses the low 8 bits, as the 8-bit STENCILTESTVAL does.
• A shader that exports the stencil reference does not get EarlyFragmentTests.
• The device enables VK_EXT_shader_stencil_export when it has it. Without it the recompile throws with the message above; there is no fallback.
• The host value is also what VK_STENCIL_OP_REPLACE writes, so with the bit set a stencil operation STENCIL_ONES or STENCIL_REPLACE_OP on a writable stencil throws (the decode folds those into the static reference, which the shader now replaces), and so does a STENCIL_CLEAR_ENABLE draw. STENCIL_REPLACE_TEST, the comparisons and the increment, decrement, invert, zero and keep operations are unchanged.
• The flag is part of the pixel stage info, the recompile cache key, the draw keys and the serialized request (version 17).

@fisherjoey, this alone will not help on your RTX 5070 Ti. To see what a path without the extension would have to cover, could you post these registers for the draw that stops: 0x1c4 (SPI_SHADER_Z_FORMAT), 0x200 (DB_DEPTH_CONTROL), 0x10b (DB_STENCIL_CONTROL), 0x10c and 0x10d (DB_STENCILREFMASK)? If you can run PPSA01341 on an AMD or Intel GPU, the next message from there would help too.

Tested

Linux (CachyOS, GCC, RTX 4090, NVIDIA driver 610.57.04; lavapipe from Mesa 26.2.3 for the device tests), Release, configured with the CI flags (-DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON), on main d06c065:

• agc_driver_graphics (new stencilRefExportTests):
  • DB_SHADER_CONTROL 0x52, 0x12, 0x02 and 0x03 decode and pass the precheck with Z formats 2 and 9, and are refused with 0, 1, 3 and 7; bit 2 is refused.
  • The recompiled module has the capability, the extension, the execution mode, the built-in and a store to it when the bit is set, also next to a depth export, and none of them when it is clear, directly and after a serialization round trip. It passes ValidateShaders, has no EarlyFragmentTests, and the cache keys differ.
  • Without the capability the recompile throws with the VK_EXT_shader_stencil_export message.
  • STENCIL_ONES, STENCIL_REPLACE_OP and a stencil clear draw throw with the bit set.
  • On main it fails with depth export, shader coverage or ordered fragment execution is unsupported (register 0x203 = 0x00000052).
• agc_driver_stencil_ref_export (new): a pixel shader exports the pixel column as the stencil reference with always-pass and replace; four draws without an export then test the stencil for equality with a column and must cover exactly that column. The exported column tested for equality against the stored one covers everything, the exported row only the diagonal, and the same shader without the bit behaves like a plain one.
  • lavapipe: passes.
  • RTX 4090: skipped, the device lacks VK_EXT_shader_stencil_export (tools/check_skips.py accepts the message).
  • On main it skips on both devices, because main never enables the extension. With the skip check removed it fails on lavapipe: stencil equal to 0 after the exported column was stored: pixel (1, 0) is 255, expected stencil-rejected.
• agc_shader_memory: the request serialization tests cover version 17 and the new flag in the shader identity.
• ctest -R "agc|shader": 443 of 443 pass on the RTX 4090 (the new device test skipped as above) and on lavapipe (35 skipped for missing device features, the same 35 as on main).
• python3 tools/check_conventions.py --base origin/main, tools/check_test_registration.py and tools/tests/test_check_skips.py pass.
• A 2D title we test with: one unattended run to gameplay passes its frame checks, no skipped draws, no GPU errors in the kernel log. It does not use the new path.

Not tested: PPSA01341 (I do not have the title); a draw with a stencil reference export on NVIDIA, AMD or Intel hardware (only lavapipe ran it); a console (the behaviour rests on the Mesa sources above); Windows (the CI job is the test).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes (Claude Code)

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#3988	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1630-1631, L1633
#4011	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1630-1631, L1633
#4114	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1633 core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L737, L765
#4163	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1633 core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L737, L765
#4172	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1633 core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L737, L765
#4201	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1633 core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L737, L765
#4210	core/libs/prx/libSceAgcDriver/Graphics/src/ShaderInputState.cpp L144, L166 core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp L188 docs/dev/TechnicalDebt.md L552
#4222	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1628, L1633 core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L737, L765