fix(shader): read the ancillary render target index from Layer when the draw exports it

[TideOPL]

What

#4842 made the render target index in the packed pixel ancillary word (bits 16 and up) the constant 0, because no pre-rasterization stage wrote Layer and reading the unwritten builtin drops the fragment's writes on AMD's Windows driver (#4800). Its description noted that once layered draws export Layer, those draws need the real value. They do now: a layered draw ORs PA_CL_VS_OUT_CNTL bit 18 into the vertex request (State::layerExports), and #3763 renders layered draws into 3D color targets. A pixel shader in such a draw reads 0 for every layer.

• Read Layer when the draw exports it, 0 otherwise. The layer field is Layer * LayerExported, where LayerExported is a new pipeline specialization constant (PipelineSpecialization::LayerExported), handled like feat(shader): give reads the masked-select pass keeps the whole pixel ancillary word #4031's RasterPrimitive: the variant records that it reads it (ShaderInfo::layerExported), and the module key and specialization carry ShaderPixelStageInfo::layerExported. With the constant at 0 the driver folds the load away, which AMD Windows: draws whose pixel shader reads the packed ancillary VGPR leave pixels unwritten (agc_driver_relative_ancillary fails on main) #4800 showed leaves AMD's writes intact ("gl_Layer loaded but unused: writes land").
• Both lowerings. lowerPackedAncillary (field reads) and ComposePackedAncillary (whole-word reads, which composed only the primitive class and the sample index) both use it.
• Plumbing. DrawDecode sets layerExported from State::layerExports for a pixel program that reads the ancillary word. The request serializer writes it from version 23 (main already uses 22 for the tessellation patches per group); older requests read false. The shader disk cache format moves to 29.
• The shader.md debt entry for the packed ancillary word now describes the two cases.

Tested

• System: Windows 11 (26200), Ryzen 7 7800X3D, RTX 5070 and the CPU's Radeon iGPU (ANYPS5_GPU=Radeon), WinLibs GCC 15.2, Release, BUILD_TESTING=ON, rebased on main 5d7f1ff.
• New case in agc_driver_layered_volume_draw: the layered volume draw again, with a pixel shader that writes (ancillary >> 16) * 0.5 to every channel, so the bands of layers 0, 1 and 2 must read 0, 128 and 255. On main (with the one layerExported line of the test removed) it fails: slice 2 texel (16, 0) channel 0 is 0, expected 128 (the band of layer 1). With this change it passes, also in the _vertex and _vertex_no_gpl variants.
• pixel_ancillary: rows with the layer exported; field reads and the composed word now include the layer only when exported, and the composed word must use LayerExported.
• agc_shader_memory: version 23 prefix, version 24 is refused, the flag round-trips, versions 19 to 22 read no flag, a truncated flag is refused. Reading the flag at version 22 fails the test.
• Full suite: ctest -E windows_backtrace: 1077 of 1077 pass on 5d7f1ff (7 skips as on main).
• AMD: on the Radeon iGPU (RDNA 2, 0x164e), agc_driver_relative_ancillary (the AMD Windows: draws whose pixel shader reads the packed ancillary VGPR leave pixels unwritten (agc_driver_relative_ancillary fails on main) #4800 reproduction), agc_driver_pixel_ancillary_word and the agc_driver_layered_volume_* tests, including the new case, pass.
• Minecraft Dungeons II (PPSA16064): it builds its 32x32x32 colour-grading LUT with one layered draw whose pixel shader takes the blue input from the layer. On a main-based build every slice was graded for blue 0, so the whole scene lost its blue channel (average blue 0 in a gameplay screenshot, purple flowers yellow). With this change the colours match our older build (blue/green ratio 0.889 against 0.87). This is not a clean-main run: the title needs other open PRs to reach gameplay.
• tools/check_conventions.py --base origin/main passes.
• Not run: Linux, macOS, AMD's discrete GPUs.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• AI-assisted: yes. Written with AI assistance (Claude Code); I reviewed and tested it.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#5200	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1846-1847, L1849-1850, L1852, L1962-1967 core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L337, L387, L753, L785 core/shader/recompiler/PipelineSpecialization.hpp L29 core/shader/recompiler/Recompiler.hpp L152

[Amoenus]

Thanks for moving layer to request23. Our staged fork heada5a282 uses your actual de5279bc commits and latest #5137 240411d, rebased on pinned main b0d with authorship preserved, followed by alpha's dither24 field. Upstream tessellation22 remains intact.

Independent untouched-author emission confirms twelve canonical23 cases; our registered test-loop bytes match all twelve exactly, including tessellation63/F16/layer combinations. Compatibility20 repetitions, full build and NVIDIA checks pass. AMD outcomes remain disclosed in the maintained alpha report.

The #4451 PR head stays unchanged until #5097/#5137 merge, per review; the final rebase will use the resulting main. Earlier layer22/dither23 proposals were superseded by #4114's upstream22 allocation. AI-assisted: Codex.

[johnmaia]

Test report: fixes Epic Mickey: Rebrushed (PPSA15657) on Linux, RX 7600, RADV Mesa 25.2.8.

On main without this PR: the title's levels render with almost no blue. Mean blue in the game area is 0.6 out of 255, while red and green reach 60–70, so the image is yellow-green. The HUD is unaffected. Cause:

• UE 4.27 builds its colour-grading LUT as a 32×32×32 volume with one layered draw (PostProcessCombineLUTs.usf, CombineLUTsCommon).
• That shader takes the LUT's blue coordinate from the layer index the pixel shader reads.
• After fix(shader): read the ancillary render target index as 0 #4842 that read is always 0, so every slice gets blue = 0, and the tonemapper maps all blue to 0.

What I checked:

• Reverting fix(shader): read the ancillary render target index as 0 #4842's merge on main restores the colours: in-level mean blue goes back to 49–82.
• Main 3faa9cb + this PR (plus fix(agc): preserve array slice XOR in color target views #4551 and fix(agc): store a storage image whose last block row runs past its allocation #5030 for other Mickey issues): the level renders with correct colour, in-level mean RGB about (18, 19, 49), with no errors in a 4-minute scripted session.
• ctest: 1050 of 1051 passed. The failure is guest_math (abort without output); this PR doesn't touch that code, so I take it to be unrelated.

🤖 Generated with Claude Code

[adriadam10]

Checked the logic (Layer * LayerExported in both lowerings, key/specialization plumbing, debt entry): it matches what the tests and the reports from @johnmaia (Epic Mickey, RX 7600, RADV) show. agc_shader_memory, agc_driver_layered_volume_draw, agc_driver_pixel_ancillary_word and pixel_ancillary pass on the PR merged onto current main.

One thing to fix:

• core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp:389: the new layerExported byte is read at version >= 22u, but main already writes and reads version 22 (writeRequest writes 22u, patchesPerGroup is read at >= 22u). A version 22 request produced by main has no such byte, so it is misread (the same kind of break as the version 18/19 case fixed in #4031). Move the flag to version 23: bump the written version and the upper bound, read it at >= 23u, and make the legacy-request test in ShaderMemory.cpp (the loop over 19..21) cover 22 as well. #5137 and #4451 need to be re-based on that number.

[TideOPL]

@AlxcBTW Tag following request

[AlxcBTW]

@TideOPL thanks for the tag. Sword of the Sea (PPSA25344, UE5) very likely takes the same path: UE's tonemapper builds its colour-grading LUT as a layered 32³ volume draw, and on main our scenes come out green-tinted. I'll run this PR on SotS today (RTX 5070, Windows) and post the blue/green ratio of the same frame with and without it.

[TideOPL]

Thanks, moved the flag to request version 23 (de5279b): the writer and the upper bound are 23, layerExported is read at >= 23u, and the legacy loop in ShaderMemory.cpp now covers 19 to 22. Reading the flag at 22 fails that test. Rebased on 5d7f1ff; ctest -E windows_backtrace 1077/1077. This matches the layer23 / dither24 order Amoenus proposed for #4451.

[adriadam10]

Re-checked on the head merged onto current main: Layer * LayerExported in both lowerings, key/specialization/disk-cache plumbing, request version 23 (>= 23u, versions 19-22 read no flag) and the debt entry. agc_shader_memory, agc_driver_layered_volume_draw, agc_driver_pixel_ancillary_word and pixel_ancillary pass; without the change the tests don't build. Also confirmed by @johnmaia on Epic Mickey (RADV, RX 7600). The only failure in the scoped run, agc_driver_tessellation, segfaults the same way without this PR. Merging once the full Build is green.

[AlxcBTW]

@TideOPL a follow-up from Sword of the Sea, and a correction to my guess above. With this PR ported onto our lab line (RTX 5070 Laptop, Windows):

• Desert gameplay: mean B/G is 0.877 with the PR against 0.824 without (R/G 1.13 against 1.30), so a bit less orange. These are not pixel-identical frames (different moment and resolution), so treat it as directional only. Neither side shows a missing blue channel.
• Opening cutscene: this is where our green tint actually is, and it is unchanged with the PR. It shows as a green/white wash with white rectangular blocks, which is a different bug on our side (a tile/clear corruption we're chasing separately), not the LUT layer.

So I can't claim a fix on SotS. There was no regression either: the session ran normally with the PR in. Sorry for pointing at the wrong cause earlier.