fix(agc): drop layer I/O when linking mesh pipelines in driver to prevent AMD crash

[Feuris]

What

Sanitize mesh layer outputs and unconsumed fragment layer inputs in Pipeline.cpp prior to vkCreateShaderModule while keeping the shader recompiler's mesh BuiltInLayer export intact.

In the proprietary AMD Windows Vulkan driver (amdvlk64.dll), declaring gl_Layer as an output from a mesh shader marks the interface variable as PerPrimitiveEXT. When linking against a fragment shader, AMD's stage matcher (0x1821f30d0) fails to match the inputs, leaving a null pointer and triggering an unconditional dereference at amdvlk64.dll+0x21f3221 (mov 0x4(%r9), %r10d).

Reverting the changes in SpirvModuleEmitter.cpp ensures that the recompiler emits spv::BuiltInLayer as expected, passing agc_driver_layered_volume_draw. Sanitizing the SPIR-V at the Vulkan driver level in Pipeline.cpp via DropMeshLayerOutput and DropFragmentLayerInput avoids the AMD crash during vkCreateGraphicsPipelines.

Tested

• agc_driver_layered_volume_draw, agc_driver_mesh, and agc_vertex_layer_export all pass 100%.
• Super Meat Boy 3D (PPSA31162): boots and renders continuously past the layered volume draw on AMD Radeon RX 6700 XT with zero crashes.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA31162
• GPUs tested: AMD Radeon RX 6700 XT (Windows 11, driver 32.0.21045.5002)
• AI-assisted: yes

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6262	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp L345, L347-348
PR 6292	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp L10
PR 6436	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp L345, L347-348

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 5609	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 5925	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6465	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6539	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6612	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6615	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6683	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6947	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6970	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 6995	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 7002	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 7201	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 7332	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp
PR 7642	core/libs/prx/libSceAgcDriver/Graphics/src/Pipeline.cpp

[Neithennn]

Windows NVIDIA run: this change fails agc_driver_layered_volume_draw, which passes on main.

Windows 11 Pro 26200, NVIDIA GeForce RTX 4080 (driver 617.42), AMD Ryzen 7 5800X, WinLibs GCC 15.2.0 r7, Ninja, Release, bundled dependencies, BUILD_TESTING=ON, ANYPS5_NO_SHADER_CACHE=1. Head 17d8442ae merged onto main ff23657f3; full build succeeds.

AGC graphics: the mesh program declares no Layer output

• agc_driver_layered_volume_draw: fails 7/7 with this PR (targeted, broad and 5 serial reruns); passed in the baseline run on unchanged main ff23657f3 on the same machine.
• agc_driver_mesh, agc_vertex_layer_export, agc_driver_layered_volume_submit, agc_driver_layered_volume_vertex, agc_driver_layered_volume_vertex_no_gpl, agc_driver_array_layer_target, agc_driver_depth_plane_guest_layers: pass.
• ctest -R "agc_|shader|recompiler|spirv|video_out" -j 8: 620/623. Besides agc_driver_layered_volume_draw: agc_driver_store_over_untracked (Windows: StoreOverUntracked fails at "the first result was not stored" #6347/AGC: StoreOverUntracked's NeighbourCopied case intermittently misses its pending result on Windows / RTX 4080 #6383, also on main) and agc_driver_multisample_fill_host_only, which failed once under parallel load and passes 5/5 alone.

The assertion is not NVIDIA-specific: with the Layer output no longer declared as a mesh stage output, a mesh program that writes the layer has no Layer output, which this test requires. Expected on every vendor, but only measured here.

AI-assisted: yes (runs done with Claude Code).

[PYTHONRETARDE]

AMD Windows run (RDNA 3): the change fails agc_driver_layered_volume_draw here too, as on the RTX 4080 above.

RX 7800 XT (1002:747e), driver 32.0.31041.1004, Windows 11 Pro 26200, WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON. Baseline main 03c4cb5e7 + #6452 (needed to build on Windows); test build adds this PR's head 17d8442ae. %LOCALAPPDATA%\AMD\VkCache emptied before each run (see #4768).

Test	Baseline	With this PR
agc_driver_layered_volume_draw	3/3 pass	3/3 fail: the mesh program declares no Layer output
agc_driver_mesh, agc_vertex_layer_export	3/3 pass	3/3 pass
agc_driver_layered_volume_vertex, _submit	3/3 pass	3/3 pass

Full CTest with the PR: 1293 of 1298 pass. The other failures also occur on the baseline, so they're not from this PR: agc_driver_quad_pixel_mask (fails 5/5 on the baseline), agc_driver_depth_surface_from_storage (fails 1/5 on the baseline), agc_driver_layered_volume_vertex (#4768 cache swap; passes 3/3 alone), guest_kqueue (#6173).

So dropping the Layer interface variable loses the layer the mesh program writes. I couldn't test PPSA31162, so I can't say whether the crash it fixes reproduces on RDNA 3.

AI-assisted: yes (Claude Code).

[Feuris]

Reverted the change in SpirvModuleEmitter.cpp to keep the shader recompiler's mesh BuiltInLayer export intact, resolving the regression on agc_driver_layered_volume_draw.

To resolve the AMD proprietary driver crash (amdvlk64.dll+0x21f3221 null pointer dereference during stage interface linking), mesh layer outputs and unconsumed fragment layer inputs are sanitized in Pipeline.cpp before vkCreateShaderModule.

Verified:

• ctest -R "agc_driver_layered_volume_draw|agc_driver_mesh|agc_vertex_layer_export" -V passes 100% (3/3).
• python tools/check_conventions.py passes cleanly.
• Super Meat Boy 3D (PPSA31162) boots and renders through the layered volume draw on AMD host (RX 6700 XT) with zero crashes.

[Neithennn]

Windows NVIDIA rerun of the new head: agc_driver_layered_volume_draw passes again.

Windows 11 Pro 26200, NVIDIA GeForce RTX 4080 (driver 617.42), AMD Ryzen 7 5800X, WinLibs GCC 15.2.0 r7, Ninja, Release, bundled dependencies, BUILD_TESTING=ON, ANYPS5_NO_SHADER_CACHE=1. Current main 72c5eed1f does not build on Windows here without #6452, so the base is main + #6452 (4fe92c9c2); head 04ab941ff merged on top.

• agc_driver_layered_volume_draw: passes (targeted and 5 serial runs); it failed 7/7 on the previous head.
• agc_driver_mesh, agc_driver_mesh_index_cache, agc_vertex_layer_export, agc_driver_layered_volume_submit, agc_driver_layered_volume_vertex, agc_driver_layered_volume_vertex_no_gpl, agc_driver_array_layer_target, agc_driver_depth_plane_guest_layers: pass.
• Full ctest -j 8, excluding agc_shader_helpers: 1301/1302; the failure is guest_kqueue (0xc0000409, test(libkernel): deny the kqueue parent listing to the token user's SID #6173), which also fails on the base alone.

agc_shader_helpers is excluded because it hangs intermittently on this machine both with and without this PR (base alone: 3 of 10 runs hit a 60 s limit; it has no timeout of its own, and one full run waited on it for 38 minutes). It looks like a separate problem and is not attributed to this change.

No AMD GPU here, so the amdvlk64.dll crash this PR targets is not tested.

AI-assisted: yes (runs done with Claude Code).

[milescoviello]

A cross-vendor check, since the description lists AMD/Windows only: NVIDIA GeForce RTX 4070 Ti (595.99.02), Ubuntu 24.04, Release. This PR's head (c857ea6057) merged onto main 661a111, clean merge:

• agc_driver_layered_volume_draw passes 3 of 3.
• Super Meat Boy 3D (PPSA31162), pad input, 150 s, 2 runs on main and 2 with this PR: all four reach the limit with no driver failure lines. The scene the runs reach depends on input timing, so I'm not comparing frame rates.

No regression on NVIDIA as far as these go.

AI-assisted: yes (Claude Code)

[thuanquang]

@Feuris I hit the same amdvlk64.dll+0x21f3221 crash on the same RX 6700 XT and driver (32.0.21045.5002) while writing the layer test asked for on #7374. I found a way around it that keeps the layer: declare the mesh gl_Layer as a member of a PerPrimitiveEXT Block struct (the way glslang declares it inside gl_MeshPerPrimitiveEXT) instead of as the standalone PerPrimitiveEXT array. With that, AMD's driver creates the pipeline and each primitive lands in its own slice, also when the pixel shader reads the layer, which is the case DropFragmentLayerInput covers.

Branch: thuanquang/AnyPS5 fix/mesh-layer-block (c99657f). It changes the emitter's declaration, teaches ShaderValidation about PerPrimitiveEXT member decorations, and adds agc_driver_mesh_provoking_layer:

• one quad whose triangles carry per-vertex layers (0, 0, 1), drawn into a 2-slice target;
• first and last provoking vertex mode, each with a white pixel shader and with one that writes the layer it receives;
• all four cases pass on the RX 6700 XT; on main the first draw crashes;
• the full ctest is otherwise unchanged.

With #6379 the layer output is removed from every mesh pipeline, so layered mesh draws always land in slice 0, and this test's last-mode case would fail. Would you take the block form into #6379 instead of dropping the layer? Or should I open it as a separate PR? Super Meat Boy 3D (PPSA31162) would be worth a rerun with it; I don't have the game.

[thuanquang]

@Feuris I opened the block form as #7674, rebased on current main, so you and the reviewers can compare the two. It keeps the layer, which #6959's layer windows need: on this RX 6700 XT, #6959's agc_driver_layered_volume_mesh crashes at amdvlk64.dll+0x21f3221 and passes with #7674 on top (run).

[oneandonlydean]

Thanks for tracking down the AMD mesh-shader crash. We're going with #7674 for it: that one keeps the mesh layer output and declares it the way glslang does, so layered mesh draws still land in the right slice on every vendor, while removing the layer here would send them all to slice 0. Closing this as superseded by #7674. If you hit an AMD case that #7674 doesn't fix, please reopen or open a follow-up on top of it.