feat(recompiler): plan waterfall loops over per-lane T#s as non-uniform tables

[fdfawef324]

What

Image accesses inside a waterfall loop over a per-lane T# failed planning with GetImageResource dword 0 is not a valid runtime value; chain: ReadLane(SelectU32(..., CompositeExtractU32x4(LoadBufferU32x4(...)), ...), ...). The loop is the one compilers emit for a divergent resource: pick a lane (s_ff1_i32_b32/b64 of the remaining mask, or v_readfirstlane_b32), copy that lane's eight T# VGPRs to SGPRs with v_readlane_b32, narrow EXEC with v_cmpx_eq_u32 to the lanes whose eight VGPRs equal the copy, run the image instruction with the SGPR T#, remove those lanes (s_andn2), repeat. The SGPR T# comes from VGPRs, so no descriptor source could evaluate it.

This plans such accesses as one access per lane with the lane's own T#:

• Waterfall fold (ResourceTracker, before descriptor-phi splitting). For an image read, sample, gather or query, every T# dword that is a ReadLane/ReadFirstLane of a value x is replaced by x when the EXEC its results are written under (the condition of every SelectU32 that takes its result) implies ReadLane(x) == x. The facts are collected from LogicalAnd/LogicalNot/LogicalOr of IEqual32/INotEqual32, and of IEqual64 of two CompositeConstructU64 (the v_cmpx_eq_u64 form). Lanes outside that EXEC still run the SPIR-V access (the s_cbranch_execz around it is wave-uniform) but their results are discarded, so which T# they use does not matter; stores and atomics are not folded. The fold is kept only when it produces something the tracker can bind: dwords that are all valid runtime values, or the table below. Otherwise the handle is left as it was, so planning that worked before (ReadFirstLane of a phi of uniform T#s through the descriptor web) is unchanged.
• Per-lane table. When the folded T# is eight dwords of buffer_load_dword* with IDXEN (immediate VOFFSET and SOFFSET) from one V# that the SRT walker evaluates, at consecutive offsets, the access becomes a bindless image table (DescriptorSource::IndirectImage) keyed by the lane's buffer index (laneKey, indexed). The loads stay; they also feed the loop's compares. A lane that the load skipped (the T# VGPRs are a masked SelectU32) gets key 0xffffffff, which is never mapped, and reads zeros. This is exact when the VGPRs it keeps are zero (a null T#). If they hold another descriptor or another table read, directly or through masked writes and phis that all eight dwords share (a default T# copied into the VGPRs under EXEC, or merged at a branch), the fold is refused and planning throws as before; any other stale value reads zeros and is recorded in docs/dev/debt/shader.md.
• Implicit derivatives. Every lane runs the access with its own record, and the record's runtime image mode (numeric class, conversion, view dimension) picks the SPIR-V path per lane. Lanes of one quad whose records need different modes would take different paths around an implicit-LOD sample or an LOD query (image_get_lod), where SPIR-V leaves the derivatives undefined, so such a table throws at dispatch with bindless image table keyed per lane holds images of different runtime modes under an implicit-derivative access, which is not implemented. Only bound records take part in the check, so a null record 0 is not compared. Lanes whose record is null or outside the table take the zero path of the per-lane mode switch under record 0's mode, the first mode when record 0 is null, so around such an access the derivatives of quad neighbours whose records need another mode are still undefined; this gap is recorded in docs/dev/debt/shader.md. Accesses of one table share the check; reads, gathers and explicit-LOD samples of a table nothing derives through still take mixed modes.
• Materialization. An indexed table's record k is at base + entryOffset + k * stride and the table has num_records records, matching the structured buffer addressing of the load. A V# with num_records 0, a null V# included, binds no record and is not checked further: with no records every OOB_SELECT puts each load out of range, so no lane can hold a record, every lane that runs the access reads zeros, and a dispatch over an unbound table whose waterfall no lane enters does not throw. A V# with records and swizzling, ADD_TID, OOB_SELECT 2 or 3, or a stride smaller than the record offset plus 32 bytes throws at dispatch.
• SPIR-V. A lane-keyed table's slot index, image and sampled image are decorated NonUniform even where wave-uniform keys are not (one wave held by one host subgroup), since the slot differs per lane.

Sources: AMD RDNA 2 ISA reference guide, chapter 12: V_READLANE_B32 (VOP3 opcode 864; lane selected by SSRC1, ignores EXEC), V_READFIRSTLANE_B32 (VOP1 opcode 2), S_FF1_I32_B32/_B64 (SOP1 opcodes 19/20), V_CMPX_EQ_U32 (VOPC opcode 210; writes EXEC), V_CMPX_EQ_U64 (VOPC opcode 242), S_ANDN2_B32/_B64 (SOP2 opcodes 20/21; SCC = result != 0), S_CBRANCH_SCC1 (SOPP opcode 5) and IMAGE_GET_LOD (MIMG opcode 96). Range checks with num_records 0: the GFX10 OOB_SELECT table in Mesa's src/amd/common/ac_descriptors.c (0: index >= NUM_RECORDS or offset >= stride, 1: index >= NUM_RECORDS, 2: NUM_RECORDS == 0, 3: offset >= NUM_RECORDS). The loop shape is LLVM's emitLoadScalarOpsFromVGPRLoop/loadMBUFScalarOperandsFromVGPR in llvm/lib/Target/AMDGPU/SIInstrInfo.cpp (readfirstlane of each operand piece, V_CMP_EQ_U32/_U64 against the VGPRs, S_AND_SAVEEXEC, the instruction, S_XOR EXEC, S_CBRANCH_EXECNZ), and AMDGPURegisterBankInfo::executeInWaterfallLoop for GlobalISel. Mesa's nir_lower_non_uniform_access.c builds the same loop for RADV before ACO for NonUniform descriptor indices; this PR undoes it in the other direction, from the guest loop back to a NonUniform access. Derivatives: the Vulkan specification's derivative operations leave implicit derivatives (implicit-LOD samples, OpImageQueryLod) undefined when the invocations of a quad are not in the same control flow. The fold is a logical consequence of those semantics; nothing here was measured on hardware.

Limits (recorded in docs/dev/debt/shader.md): only IDXEN loads from an evaluable V# are planned; waterfalls over other per-lane T#s, over S#s, and around image stores or atomics still throw. The equalities are read from the EXEC lane condition the translator builds for v_cmpx and, in wave32, for s_and_saveexec of a v_cmp/VCC result; compares combined with s_and_b32 in other SGPRs, and s_and_saveexec_b64 in wave64, leave EXEC as a mask, so those loops still throw (not tested). On main a per-lane table goes through the whole-table mode of main's bindless table (there are no material records to enumerate its keys from), so any table V# with more than RuntimeAbi::BindlessTableSlots (16) records is rejected at dispatch with bindless image table has N entries (0 materials), limit 16. #758 (stacked on #757) lifts this with per-invocation table snapshots; whichever lands second ports the lane-keyed table to the other's model (#757 rewrites DescriptorSource::IndirectImage, TryMakeTableImage, resolveTableImage and the SPIR-V table selection that this PR extends).

Merged #4883 persists resource plans. This branch now encodes and decodes all three additional IndirectImage fields (laneKey, indexed, implicitDerivatives), retains the strict ten-field guard, and uses fresh disk format 31 after main format 30. Fresh and loaded plans must preserve indexed table addressing and derivative-mode checks.

Tested

Windows Server 2025, MinGW-w64 GCC 15.2.0, NVIDIA GeForce RTX 3060 (subgroup 32). The first two commits were tested on main c8eb1f5; the third (empty tables, bound modes) and the rebase on main 2f6ae9f as noted below.

• New agc_driver_waterfall_images (tests/execution/WaterfallImages.cpp): hand-assembled compute shaders that load a per-lane T# from a 4-record table with two buffer_load_dwordx4 ... idxen (key tid & 3, one record a null T#, records of an R32_UINT and an R32_SFLOAT texture) and run the waterfall loop around image_load at a per-lane coordinate; three dispatches with different tables must reuse one compiled artifact, and the SPIR-V must decorate the table slot NonUniform. Variants that must read each lane's texels: wave32 over the whole wave, wave32 under a partial EXEC (lanes skipped by the load stay 0), wave64 under a partial EXEC (when the host subgroup holds 32 lanes), a loop that picks lanes with v_readfirstlane_b32, and a loop that compares the T# with four v_cmpx_eq_u64. Variants that must still throw: lanes whose T# VGPRs keep another descriptor, lanes whose T# VGPRs got a descriptor written under EXEC, a loop that compares seven of the eight dwords, an image load after the loop restores EXEC, and an image store in the loop. A loop behind s_cbranch_execz over the lanes whose key is below the table V#'s num_records must read each lane's texels over the 4-record tables, and must neither throw nor change the value each lane holds before the loop (-1) over a null table V#, a V# with stride 0 and no records, and a swizzled ADD_TID V# with OOB_SELECT 2 and no records. An image_get_lod waterfall compiles over a table of R32_SFLOAT records and over a null record 0 with R32_UINT records, and must throw over a table that mixes R32_UINT and R32_SFLOAT, with or without a null record 0.
  • Without the change (sources at main, the test kept): fails with ShaderRecompiler::Recompile: GetImageResource dword 0 is not a valid runtime value; chain: ReadLane(CompositeExtractU32x4(LoadBufferU32x4(GetBufferResource(GetUserData....
  • With the SPIR-V hunk reverted: fails with wave32 waterfall over the whole wave: the per-lane table slot is not decorated NonUniform.
  • With the skipped-lane check limited to the values themselves (not their masked writes and phis): fails with a waterfall over a table T# and a T# written under EXEC to the VGPRs of lanes the load skipped was accepted; dispatched, lane 7 of that shader read 0 instead of texel 4097 of the default texture.
  • Without the runtime-mode check: fails with an LOD query in a waterfall over a table of float and integer images was not refused.
  • Without the third commit's fix (its test kept, each case run on its own): the null table V# and the stride-0 V# fail with ShaderRecompiler::Recompile: bindless heap record is smaller than its descriptor, the swizzled ADD_TID OOB_SELECT 2 V# with bindless image table indexed through a V# with swizzling, ADD_TID or an unbounded range check is not implemented, and the null record 0 with R32_UINT records with bindless image table keyed per lane holds images of different runtime modes under an implicit-derivative access, which is not implemented; the mixed table with a null record 0 is refused with and without the fix.
  • With the change: passes, 20 repeated runs (on 2f6ae9f with all three commits).
• ctest, 69 related tests, all pass: agc_shader_memory, agc_shader_bindless, agc_shader_srt_read_patching, agc_shader_frontend_pair(_off), agc_structurizer*, agc_spirv_module, agc_masked_select, agc_shader_disk_cache, agc_shader_control_flow*, agc_driver_bindless_images, agc_driver_buffer_descriptor_runtime, agc_driver_exec_host_lanes, agc_driver_read_lane, agc_driver_runtime_images, agc_driver_wave32_subgroup, every agc_driver_image_* and every agc_driver_prepared_* test, and agc_driver_waterfall_images (first two commits). With the third commit, rebased on 2f6ae9f: agc_driver_waterfall_images, agc_shader_bindless, agc_shader_memory, agc_driver_bindless_images, agc_driver_buffer_descriptor_runtime and agc_driver_runtime_images pass.
• python tools/check_conventions.py --base upstream/main (upstream/main 2f6ae9f, all three commits): passed.
• Offline replay (agc_shader_replay, first two commits) of 139 captured requests from six titles, main 2a2a3dc and this branch on it, both the recompile and the registration (--prepare) path: 132 requests give the same result on both builds (101 compile to the same number of SPIR-V words, 31 fail with the same message); only the seven GTA V requests below change.
• Grand Theft Auto V (PPSA04264), offline replay of the captured requests only (no run in game): the pixel shader 0x25cb2dc00 that stopped the title at a draw and six compute shaders refused at registration (0x25cbd6c00, 0x25cbe7800, 0x25cbf0000, 0x25cc00a00, 0x25cc08400, 0x25cc0ac00) fail planning on main, on feat(agc): implement image tables loaded through 64-bit pointers #758 (rebased on main 3779597) and on feat(agc): image-table records that hold a pointer to the T# or split it over two loads #4475 (the six compute requests; its serializer cannot read the pixel request) with the error above. With this change all seven pass planning and the registration path compiles them (259514, 975997, 1812923, 1918841, 469346, 696742 and 709414 SPIR-V words, in the order above; the pixel shader and 0x25cbd6c00 give the same counts after the rebase). The requests carry zero user data, so the recompile path, which now has to read the table V# to materialize the table, stops later at SrtWalker::EvaluateDescriptorSources (the SRT pointer is null); for the same reason the table contents are not in the requests, so whether the tables fit in 16 records, and whether the table behind the pixel shader's image_get_lod waterfall holds records of one runtime image mode, was not checked.

Not tested: in game, on lavapipe/Linux, on AMD hardware or RADV, console behaviour.

Latest source rebase: main 0df6e295c, with original commit authors/messages and complete rebased trees independently verified. The added codec compatibility commit is tested with the exact public production sources and matching IR units in a CPU-only fixture: all eight combinations of the three flags have distinct encodings and round-trip correctly. Removing only those field reads/writes fails the encoding assertion; changing only format 31 back to 30 fails the old-entry refusal assertion. The latest rebase preserves these tested production/fixture blobs exactly. This does not claim a fresh native or title run of the rebased branch.

Checklist

• Based on current main; no other open PR implements the same functions (feat(agc): implement image and sampler tables indexed at runtime offsets #757, feat(agc): implement image tables loaded through 64-bit pointers #758 and feat(agc): image-table records that hold a pointer to the T# or split it over two loads #4475 plan scalar-loaded table T#s; none folds v_readlane waterfalls or plans T#s loaded per lane by buffer_load)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none (overlaps feat(agc): implement image and sampler tables indexed at runtime offsets #757's table code; see What)
• Unblocks: PPSA04264 Grand Theft Auto V (PPSA04264) #1203 (planning of the waterfall refusal at 0x25cb2dc00 that stops the title, and of six compute shaders refused at registration; not run in game, and dispatch still needs the tables to fit in 16 records)
• GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100
• AI-assisted: yes

🤖 Generated with Claude Code

Co-Authored-By: Codex noreply@openai.com

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#5180	docs/dev/debt/shader.md L46-47

[nikowolf123]

Heads-up about a conflict that neither git nor the overlap check shows. My #4883 (open) stores the resource plan in the shader disk cache, and it writes DescriptorSource::IndirectImage one field at a time. On 3d86335, the encoder is at ShaderDiskCache.cpp L848-L856 and the decoder at L990-L999. Both are guarded by static_assert(sizeof(DescriptorSource::IndirectImage) == 28) at L104.

The three bools this PR adds (ControlFlowInfo.hpp L36-L38) fit in the padding after hasMaterial. The struct stays 28 bytes, DescriptorSource stays 104, and the guard does not fire. I checked this by compiling #4883's two size asserts against this PR's header with GCC 15.2: both pass, and the same assert with a different size fails.

#4883 uses a plan loaded from the disk cache in place of the front end (Recompiler.cpp L423-L425). If both PRs are merged, a warm-cache run gets the plan with laneKey, indexed and implicitDerivatives all false:

• resolveTableImage takes the 32-byte entry path (ResourceMaterializer.cpp L344-L345) instead of the V# stride (L338-L342).
• The mixed-mode check at L417 is skipped.

Example: a table V# with stride 48, the T# at record offset 16, and 3 records.

• Record k is read at base+16+32k instead of base+16+48k, so keys 1 and 2 read the wrong dwords.
• The 32-byte path counts (144 − 16) / 32 = 4 entries, so key 3 maps to record 2's T#. The stride path has 3 entries and leaves key 3 unmapped.

Whichever PR lands second has to add the three fields to EncodePlan/DecodePlan. If this one lands first, I will do that in #4883 and also make its guard catch fields added inside padding.

AI-assisted: written with Claude Code; I read both diffs and compiled the size checks described above. I did not build or run either PR's tests.

[nikowolf123]

Following up on the note above: #4883 now pins the field count of every structure its plan codec writes (664ad77). With this PR's three bools in IndirectImage, the build fails at IndirectImage changed: update the plan encoder (7 fields expected, 10 found) on GCC 15.2, Clang 21 and MinGW 15.2, so whichever PR lands second has to add the fields to the codec before it compiles.

[fdfawef324]

Thanks for checking this and adding the field-count guard. I agree that laneKey, indexed, and implicitDerivatives must survive the resource-plan codec; the unchanged structure size was not enough to detect that overlap.

Please keep #4883's codec guard. If #5089 merges first, your proposed codec update in #4883 covers the three fields. If #4883 merges first, I will rebase #5089, add the fields to EncodePlan/DecodePlan, advance the affected cache format, and check the stride-48 / offset-16 three-record case on both fresh and loaded plans before updating this PR. I have recorded the merge-order dependency in the description.

AI-assisted: yes.

[Tamalero]

For reference, a title this does not cover, so nobody counts on it: Guardians of the Galaxy (PPSA01748). Tried on main d8888df9b with this PR alone, its unplanned image accesses at pc 0x4378/0x4380 are unchanged. Their chain has no v_readlane copy of a per-lane T#. Instead, a T# is loaded through an SRT pointer at readfirstlane(key) << n, with the key derived from an ImageRead. That's a uniform-key pointer table (#757/#758 territory), not this PR's waterfall. Details are on #757.

One more thing for sequencing: this PR conflicts with #757 + #758 in ResourceTracker.cpp (12 hunks against their current heads), so titles that need both can't stack them yet.

Tested and written by Claude (AI assistant) on a contributor's machine.

[fdfawef324]

Thanks for checking this. Guardians' access is outside this PR's per-lane descriptor waterfall fold, and I have not tested that title. The public branches still need a merge-order composition with #757/#758; a private integration does not establish Guardians coverage.

The #4883 plan-codec overlap is now handled at 8416241: all three flags are encoded and decoded, the ten-field guard is retained, and format31 rejects older entries. The CPU round-trip and selective-red checks described in the PR do not claim a fresh combined native or title run.

AI-assisted: yes.
GPUs tested: none for this follow-up (CPU validation is described above).

Co-Authored-By: Codex noreply@openai.com

[adriadam10]

Rebased onto main 145cb82 and pushed two commits of ours:

• test(agc): read the waterfall test SPIR-V through a span: DecoratesNonUniform took const std::vector<std::uint32_t>&, and fix(shader): map retained SPIR-V from shader cache entries #4806 made Words() return std::span<const std::uint32_t>, so the test no longer compiled on main.
• fix(recompiler): move the disk format to 32 after main took 31: fix(shader): map retained SPIR-V from shader cache entries #4806 already moved FormatVersion to 31, so both sides' 30 → 31 merged silently and main's format-31 plan entries would have been decoded with the three new IndirectImage fields. feat(agc): implement image and sampler tables indexed at runtime offsets #757/feat(agc): implement image tables loaded through 64-bit pointers #758 renumber after this one.

Checked: scoped build + ctest on RADV (agc_driver_waterfall_images 20/20; agc_recorder_kept_bytes and agc_driver_tessellation fail the same way without this PR), agc_driver_waterfall_images/bindless_images/runtime_images/buffer_descriptor_runtime on CI's lavapipe, and agc_driver_waterfall_images fails with the original planning error when the source changes are reverted. The loop itself on RDNA2 hardware (gfx1035, wave32, s_ff1 and v_readfirstlane picks, 8×v_readlane_b32 + 8×v_cmpx_eq_u32 + image_load + s_andn2, per-lane T#s over two textures, a null T# and lanes outside EXEC): every lane reads its own texture, the null T# reads 0, and lanes outside EXEC keep their value, as the fold assumes. Merging.

[milescoviello]

agc_driver_waterfall_images fails on main da4d5f2 on Linux with an RTX 4070 Ti (NVIDIA 580.142), 3 of 3 runs, at the first LOD case:

AGC graphics: an LOD query in a waterfall over a table of one image mode failed: ShaderRecompiler::Recompile: SPIR-V validation before optimization failed:
level=2 word=18: [VUID-StandaloneSpirv-DerivativeGroupQuadsKHR-10151] Local Size execution mode dimensions is (X = 32, Y = 1) but Entry Point id 39 also has an ...

RunLod asks for host subgroup size min(32, device subgroup size), which is 32 here, so the shader is emitted with laneCount 1. SpirvModuleSetup.cpp then keeps the guest's 32×1×1 as LocalSize while adding DerivativeGroupQuadsKHR for the ImageQueryLod, and quad derivatives need even X and Y. Only the laneCount == 2 path reshapes the group to 2×N, and EmitBuiltinU32 derives LocalInvocationId/GlobalInvocationId from the index for that path. A device whose subgroup size is below 32 takes that path, which may be why it passes elsewhere.

Doing the same 2×N reshape (and the index-derived IDs) whenever a compute shader needs derivatives would also make each derivative quad four consecutive lanes, as on RDNA. Today a quad-compatible 8×8 group pairs lanes 0, 1, 8 and 9. I can send that as a PR if you'd like, or leave it to you since it's your test.

AI-assisted: yes

[adriadam10]

Reverted in #5824: after the merge, agc_driver_waterfall_images failed on every main build. CI builds with -DANYPS5_ENABLE_SPIRV_TOOLS=ON, and validation rejects the compute shader: VUID-StandaloneSpirv-DerivativeGroupQuadsKHR-10151 (local size 8x1 with DerivativeGroupQuadsKHR, both dimensions must be even). Please reopen as a new PR with the derivative group mode chosen from the local size, or with the test on an even 2D size, and build with that flag on.