fix(shader): drop masked selects proven unobserved through whole quad mode

[tibzejoker]

What

GTA V (PPSA04264) stops on current main (2220eff) at its first pixel shaders. Replaying the captured requests with agc_shader_replay:

shaders/shader_242d4b800.req: 152 code words, 4 user data, 2 memory regions, wave64
  recompile failed: ShaderRecompiler::Recompile: shader info collection failed: packed pixel ancillary input has an unsupported live use (used by SelectU32 ? ?)
shaders/shader_242d6ee00.req: 68 code words, 12 user data, 2 memory regions, wave64
  recompile failed: ShaderRecompiler::Recompile: shader info collection failed: packed pixel ancillary input has an unsupported live use (used by SelectU32 ? ?)

The SelectU32 writes a copy of the ancillary VGPR under a whole quad mode exec mask, WQM(X), where X is a ballot of the live lanes taken in the entry block. Its value is read only by lanes inside X, and by implicit-LOD samples. The masked-lane walk could not prove either case, so the select kept the whole ancillary word alive.

It also fixes Dead Cells (PPSA15552) pixel shader 0x21a23e700, on devices without VK_KHR_fragment_shader_barycentric, as reported by @hansjm10 below. It stopped with the emulated barycentric inputs exceed 32 locations, because a select under the s_wqm exec kept the I barycentric alive. With the change it takes the host-interpolation path.

MaskedSelectEliminator now proves both:

• WQM superset: a lane set contained in X is contained in WQM(X). Containment is proven through BitwiseAnd and through ballots evaluated in the uniform entry blocks, where every invocation is active.
• Quad-local samples: under a WQM mask the masked lanes form whole quads. An implicit-LOD pixel sample only reads its own quad, so it does not observe them.

With the change, both shaders recompile (4456 and 1840 SPIR-V words).

Tested

Windows 11, MinGW GCC 15.2.

• agc_masked_select_tests:
  • fails on main, on the new cases ("an implicit-LOD pixel sample under whole quad mode reads no masked quad and drops the select", "a write under whole quad mode read only by the lanes it was widened from drops the select");
  • passes with the change, including the negative case where other lanes read the value.
• agc_shader_replay on the two GTA V requests above: they fail on main and recompile with the change.
• python tools/check_conventions.py --base upstream/main passes.
• ctest -R "agc_driver|agc_masked" (298 tests, RTX 4090): 297 pass; agc_driver_emulated_color_compare fails the same way on main.
• RADV (gfx1036, Mesa 22.3.6), by @hansjm10: agc_driver_gated_dispatch_capture passes 13 of 13 at db1f3c38e, and the full agc_driver|agc_masked run has the same 10 failures with and without the change.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV

[adriadam10]

Thanks for the analysis, the reasoning in MaskedSelectEliminator looks sound and agc_masked_select passes here (and fails without the change). One blocker:

• agc_driver_gated_dispatch_capture fails with this PR merged onto current main (3 of 3 runs, RADV): radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery, then the gated submissions did not finish. With MaskedSelectEliminator.cpp reverted to main the same test passes (2.7 s). So a select that is observed is being dropped in that shader, or the generated code hangs. The PR text says the whole suite wasn't run; please run ctest -R agc_driver and fix.
• Possible lead: withinQuadsOf is used in Implications::Implies for every stage, while only the sample exemption (quadMasked) is limited to Pixel. Check whether the gated-dispatch compute shader reaches the new path, and add a test for that case.

[tibzejoker]

Thanks for the review. Rebased onto current main (2220eff) and checked:

• agc_driver_gated_dispatch_capture never reaches the new path. With APS5_DUMP_IR=all, neither of its two compute shaders (the spin loop and the reader) has a select left when MaskedSelectEliminator runs. The only SelectU32 (the reader's, with condition true) is folded away before the pass. The IR after srt and resources is byte-identical with and without this PR. So the pass cannot change the code this test runs. I can't reproduce the RADV hang here (no AMD GPU, only RTX 4090 / Vulkan). The test passes in about 1 s with and without the PR, both variants (_short_timeout too).

• ctest -R "agc_driver|agc_masked" (298 tests) with the PR: 297 pass. The one failure, agc_driver_emulated_color_compare, fails the same way on main without the PR, so it is unrelated (NVIDIA).

• Stage question: withinQuadsOf is stage-independent on purpose. WQM(X) ⊇ X holds for s_wqm in any stage, so a select written under WQM(X) and read only under X is unobserved everywhere. Only the sample exemption depends on quads having implicit derivatives, and that stays Pixel-only. Added tests:

  • the WQM implication drops the select in Compute as in Pixel;
  • a Compute read by a lane outside the widened mask keeps it.

  The new and existing WQM cases fail on main and pass with the change.

If the hang still shows up on your RADV box with this rebase, the shader logs from APS5_DUMP_IR=all or the replay requests would help me find which shader changes. From what I can see here, none of them do.

[MarcZX10]

Thanks for the updated IR trace and the compute regression cases. That addresses the specific hypothesis that agc_driver_gated_dispatch_capture reaches the new MaskedSelectEliminator proof on this shader. The reported 3/3 RADV failure versus 2/2 on main is still an unresolved validation result, though: can someone with RADV rerun the rebased commit and confirm whether the failure persists? If it does, the shader dump / IR-stage comparison for that run would help isolate it; if it no longer reproduces, recording the commit and test result here would close the outstanding concern.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#3550	core/libs/prx/libSceAgcDriver/tests/MaskedSelect.cpp L699 core/shader/recompiler/Optimization/src/MaskedSelectEliminator.cpp L477, L481-482, L523-524

[hansjm10]

RADV results for @MarcZX100's question. Setup: Debian 12, GCC 14.4, Release, ANYPS5_ENABLE_SPIRV_TOOLS=ON, AMD Radeon iGPU through RADV GFX1036 (gfx1036), Mesa 22.3.6, selected with ANYPS5_GPU=RADV (checked in the Selected GPU line).

• agc_driver_gated_dispatch_capture at this PR's head db1f3c38e (on main a11c8f7): 13 of 13 runs pass, and _short_timeout passes 3 of 3, at about 0.9 s each with no context loss. Both also pass inside the full parallel run below. On main a11c8f7 both pass 3 of 3.
• ctest -R "agc_driver|agc_masked" -j4 on RADV: 289 of 299 pass with this PR and 289 of 299 on main. The same 10 tests fail on both: alpha_to_coverage, bindless_images, depth_compare_copy, ds_append_consume, gds_append_consume, image_sint8, pixel_helper_lanes, read_first_lane, read_lane, shader_clock. agc_masked_select passes.

So I can't reproduce the hard recovery on this RADV. If it came from a newer Mesa or another GPU, the Mesa version from that run would tell.

A second title this fixes: Dead Cells (PPSA15552), on a device without VK_KHR_fragment_shader_barycentric (RADV with Mesa 22.3.6 here; lavapipe in CI). Its pixel shader 0x21a23e700 stops with Barycentric geometry SPIR-V: the emulated barycentric inputs exceed 32 locations. The shader reads its 11 parameters only through v_interp_p1/p2 pairs. After the last interpolation it writes v0/v1 again under the s_wqm exec, then restores the original exec before the export. HostInterpolationChecker therefore finds the I barycentric still alive through that masked select, and the shader takes the emulated path, which needs 11 × 3 + 1 locations. With this PR the dumped request replays to 15958 SPIR-V words on the host-interpolation path: 11 plain inputs, no PerVertexKHR or BaryCoord. Caveat: on main the title stops earlier, on NGS2 rack 0x4001 (#3793) and then on the Geom family; this shader is reached with those applied.

AI-assisted: yes (Claude Code).

[tibzejoker]

Thanks a lot @hansjm10, both for the RADV runs and for the Dead Cells shader (0x21a23e700). I added it to the description.

@adriadam10: with these results (13/13 on RADV gfx1036, and the same 10 failures with and without the PR across the full suite), could you rerun on your box at db1f3c38e? If it still hangs there, the Mesa version and GPU would help narrow it down.

[adriadam10]

Checked on RADV: the PR merged onto current main, scoped build and ctest of libSceAgc/libSceAgcDriver 406/406, including both agc_driver_gated_dispatch_capture variants. The earlier context loss no longer reproduces here, and @hansjm10's 13/13 agrees. agc_masked_select fails without the change and passes with it. The WQM-superset and quad-local-sample proofs fall back to "observed" on any shape they don't recognise, so the elimination is conservative. Merging.