fix(shader): put the compute wave index in TG_SIZE[11:6]

[houcineBen]

What

A compute wave with TG_SIZE_EN found its index in the workgroup only at bit 20 of the TG_SIZE SGPR. GCN and RDNA code reads it from bits [11:6]: ACO lowers load_subgroup_id to s_bfe_u32 tg_size, 6 | 6 << 16 (Mesa src/amd/compiler/instruction_selection, GFX10/GFX11 compute path). Every wave of such a shader read 0 there, so it took itself for wave 0.

The prologue now also writes the wave index to [11:6]. The bit-20 copy, the wave count in [5:0] and bit 31 are unchanged.

Seen on a RADV (Mesa 26.2) build running on AnyPS5: the radix sort's prefix sums across waves lost every carry but wave 0's, the sorted keys were garbage, and the PLOC BVH build that reads them spun forever in its task loop (host GPU hang, Windows TDR / VK_ERROR_DEVICE_LOST). With the fix the sort matches a CPU sort and PLOC completes, offline from a captured dispatch and in the title.

Test: agc_driver_tg_size_wave_id runs a 256-thread compute group in wave64 and wave32; every thread stores s_bfe_u32 TG_SIZE, 0x60006 and must read thread / wave size. It fails on main (every thread reads 0) and passes with the change.

Tested

Windows 11, NVIDIA RTX 4060 Ti, MinGW GCC 15.2.

• agc_driver_tg_size_wave_id: fails before, passes after.
• Full ctest on this branch: 1073 passed, 7 skipped (offscreen video-out and an unsupported stencil export), 0 failed. patched_libraries and windows_fixed_base failed once under -j8 (DLL load error 127 from the shell PATH, 5 s subprocess timeout) and pass when rerun alone.
• vkQuake (LibreQuake) homebrew PPSA99010: map load gets past the RADV BVH builds (~517 dispatches that hung before) and the demo plays in-game. It still needs separate fixes (shader compile time) that are not part of this PR.

Not verified on PS5 hardware; the source is the ACO lowering above.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• AI-assisted: yes

🤖 Generated with Claude Code

[iDiesIrae-Req]

Ran this against vkQuake (PPSA99010) with RT shadows on, which builds acceleration structures through the title's RADV port. Setup: Windows 11, RTX 4070 Laptop (driver 32.0.16.1714), WinLibs GCC 15.2.0 r7, Release, main 61b8742 with #5177, this PR and #5180 merged locally, plus a local change that gives unregistered compute shaders their FLOAT_MODE from the live COMPUTE_PGM_RSRC1 (described on #5180). That change gets the title past the #981 s_round_mode throw, so it reaches its RT pass.

On 3 of 3 runs it still stops after 12 presented frames: radv/ps5: the GPU has not reached submission 18 (at 17) in 10 s. It doesn't look like a GPU hang, though:

• The busy thread: one thread sits at 100% of a core for the whole run. gdb shows it inside NVIDIA's shader compiler (nvgpucomp64.dll), called from vkGetInstanceProcAddr / vkEnumerateInstanceLayerProperties frames in nvoglv64.dll, under AgcDriver::VulkanDevice::dispatch.
• No sign of finishing: after 180 s that thread had used 172 s of CPU, and the title was still on 12 frames.
• No driver error: the host reports no device loss or TDR, and fix(agc): print a GPU queue worker failure when it is first stored #5177 prints no [gpu] driver failure.

The last shader prepared before the stall is the RT compute shader 0x20bf82000 (was not registered; preparing its artifact at dispatch). So on this setup the stop is NVIDIA's compiler taking minutes over pipeline creation for that shader. The title's 10 s fence wait runs out first. This is likely the same thing nicolasfalesy saw on ANV on #1578 ("deep in recursive frames inside libvulkan_intel.so (it looks like a shader compile)").

So this PR doesn't change vkQuake's stop here. It may still be needed once the pipeline compiles, but I can't reach that point to check.

AI-assisted: yes (Claude Code); all runs were on my machine.

[houcineBen]

Thanks for testing. That compile stop is the first wall we hit too, before the one this PR fixes. On an RTX 4060 Ti the RADV acceleration-structure compute shaders took 58 s and ~530 s in NVIDIA's compiler. We got past it by letting one run finish those compiles into the pipeline cache (shader_cache/vk-pipelines*, saved periodically, so don't kill the process early). With the cache warm, the acceleration-structure builds pass in milliseconds.

The next stop was a real GPU hang (Windows TDR / VK_ERROR_DEVICE_LOST) in the PLOC BVH build 0x20136db00. Its radix-sort input was garbage, because every wave read wave 0 from TG_SIZE[11:6]. With this PR the sort matches a CPU sort, PLOC completes, and vkQuake gets past its ~517 BVH dispatches. After that it stops on a ~5 min AnyPS5 recompile of 0x20134f400 (structurizer; #5028 plus a small follow-up brings it to 0.7 s), and then the demo plays.

The bug doesn't depend on vkQuake: agc_driver_tg_size_wave_id fails on main (wave64 thread 64 read wave 0 from TG_SIZE[11:6], expected 1) and passes with this PR.

[iDiesIrae-Req]

@houcineBen Thanks, that worked. I reproduced your path on an RTX 4070 Laptop (driver 32.0.16.1714), Windows 11, WinLibs GCC 15.2.0 r7, Release. Build: main 8cf6a0b with this PR, #5180, #5028 and #5177 merged locally (all clean), plus the COMPUTE_PGM_RSRC1 float-mode fallback for unregistered compute shaders from my #5180 comment. vkQuake (PPSA99010) with the shareware pak0.pak, RT shadows on (default settings).

• Warm-up: 9 launches in a row, each kept running until the title gave up (QUAKE ERROR: vkWaitForFences failed with code -4), with the caches kept between launches. The first two launches each sat in nvgpucomp64 for about 5.5 and 4 min. Every launch after that added one or more entries: the pipeline cache grew from 3.4 MB to 12.8 MB and the AnyPS5 shader disk cache from 21 MB to 35 MB. On the 9th launch the title played through demo1 and started demo2.
• Warm run: 60 s with RT shadows on, no has not reached, no QUAKE ERROR, no [gpu] driver failure. The title's own PS5 present lines report 33-38 fps (26-30 ms per frame), against 55-57 fps with r_rtshadows 0 on the same machine. The screenshot shows E1M1 lit correctly.

One difference from your description: on this setup the title doesn't reach the acceleration-structure builds without the float-mode part. Without it, 0x20bf82000 throws on s_round_mode 8 at pc 316 (#981). With #5180 alone it throws the "shader without a declared FLOAT_MODE" variant, because that compute shader is unregistered and RegisteredFloatMode returns nullopt (details on #5180).

AI-assisted: yes (Claude Code); all runs were on my machine.

[adriadam10]

Checked the emitEntryPrologue change: wave index now also in TG_SIZE[11:6] (max 31 fits in 6 bits), wave count in [5:0], bit 20 and bit 31 unchanged, consistent with ACO's s_bfe_u32 tg_size, 6 | 6 << 16. agc_driver_tg_size_wave_id passes with the change and fails without it (wave64 and wave32). Merging once the full Build is green.