feat(shader): implement compute LDS beyond the device's shared memory limit

[Celegans12]

What

A compute shader whose LDS exceeds the host's shared-memory limit could produce an oversized Vulkan Workgroup array. This moves the whole LDS, including its atomic lock dword, into a coherent storage buffer with a separate slice for each workgroup. Programs that fit keep using shared memory.

The driver sizes and binds that buffer from the dispatch dimensions, rejects ranges above maxStorageBufferRange, and reads indirect dimensions on the CPU for spilled programs. LDS barriers and atomics use the corresponding storage-buffer memory semantics.

Full-result serialization preserves workgroupMemoryDwords, with a nonzero round-trip fixture and a 312-byte Linux RecompileResult assertion. Production disk entries already persist ShaderInfo::sharedMemoryBytes, from which the stride is reconstructed; the disk format is unchanged.

Tested

Windows 11/MinGW-w64 GCC 15.2.0 and Linux GCC 13.3, based on main 256b8ed: configure -DBUILD_TESTING=ON, full build, libs and registration pass. Tested merge: b5b072c8.

• Windows NVIDIA: 642 passed, five skipped, zero failures; all four skipped VideoOut modes also pass natively on unchanged binaries. The fifth skip is the no-Vulkan negative case.
• Linux lavapipe: 607 passed, 32 skipped, zero failures. The raw skip gate rejects only write tracking: WSL 6.6 returns UFFD EINVAL; this coverage remains missing.
• LDS, program validation and disk-cache tests pass on both. Cold LDS takes 1.5127 s with fresh Mesa/XDG caches and application caching disabled; OS page caches were not flushed.
• The stride codec fixture fails with its two operations removed, then passes restored. Production disk entries already preserve LDS size.
• An earlier-base Windows run had one F32 failure; both devices pass on this base containing revert(shader): undo #2279's fma NaN result selection #2584. No suite retries.
• Not tested: console behaviour, a new game run, cross-process oversized-LDS disk reload.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

[fisherjoey]

Linux results on two real GPUs with different shared-memory limits, since this was tested on Windows and lavapipe. Everything passes here, including the new agc_driver_lds_beyond_device_limit.

Nobara 44 (Linux 7.2), GCC 16.2.1, Release, BUILD_TESTING=ON. PR head merged onto main 29e533a.

	maxComputeSharedMemorySize	ctest -R agc_driver -j1
RTX 5070 Ti, driver 595.91.07	49152	196 / 196
RADV, Ryzen 5 7600X iGPU (gfx1036), Mesa 26.2.1	65536	196 / 196

CPU tests (-E agc_driver): 283 / 284. The one failure is guest_dynamic_loader (subprocess aborted), which fails the same way on main 29e533a.

[johnmaia]

Linux run on AMD, since the description lists it as not tested: Ubuntu 24.04.5 (kernel 6.8), GCC 13.3, Release, AMD Radeon RX 7600 (RADV NAVI33, Mesa 25.2.8). This PR's head 5cad357b, merged onto main f1451396. Full ctest: 495/495 pass, with the GPU tests running on the RX 7600 (none skipped). That includes the new agc_driver_lds_beyond_device_limit.

[adriadam10]

Thanks for the work and the Linux/AMD/NVIDIA runs. This one doesn't build on current main:

• core/shader/recompiler/ShaderDiskCache.cpp:69: static_assert(sizeof(RecompileResult) == 296 ...) fails because of the new RecompileResult::workgroupMemoryDwords (Recompiler.hpp). Rebase on main, update the asserted size, and say in the PR whether EncodeResult/DecodeResult need the field (it is derived from ShaderInfo::sharedMemoryBytes in materializeResult, so a decoded result probably needs it set too, otherwise a decoded result binds no set 1).
• Please rerun the Windows job after the rebase: agc_validate_program aborted with 0xc0000409. It doesn't touch LDS, so it may be a flake, but it has to be green.
• Please run the full ctest after the rebase and put the new agc_driver_lds_beyond_device_limit time in Tested (a cold-pipeline run on lavapipe counts against the 120 s CI timeout).

Once it builds and is green, the rest looks fine to me.

[adriadam10]

Checked on RADV and lavapipe: the PR merged onto current main, AGC/AgcDriver ctest 338/338, including agc_driver_lds_beyond_device_limit and agc_shader_disk_cache. With the device-memory LDS selection forced off the new test fails, so it pins the change. Consecutive dispatches sharing the buffer are ordered by the pre-dispatch barrier. Thanks to @fisherjoey and @johnmaia for the NVIDIA and AMD runs.

Pushed on top, docs only:

• docs/dev/TechnicalDebt.md: a Functional line for the storage-buffer LDS (not measured on hardware, slower than shared memory, indirect group counts read on the CPU).
• docs/dev/ARCHITECTURE.md: a bullet under the shader pipeline for descriptor set 1 and the per-workgroup slice.

Merging once CI is green.