feat(shader): lower full-width 2D BY2/BY4 image transfers

[kali113]

Status

Superseded by merged PR #1793. A maintainer accidentally closed this PR while rebasing, preserved my original work as the first commit of #1793, and added hardware measurements and corrections. This branch was reset to main during that incident; restoring it would duplicate the merged continuation.

AMD hardware measurements invalidated this draft's assumptions about wide formats, unaligned coordinates and bounds. The final implementation supports the four BY load variants for measured narrow formats, with aligned fetch windows and all-or-nothing bounds checks. The four store variants decode but still throw. The dashboard's eight newly decoded opcodes must not be read as eight completed runtime implementations.

Validation

The original draft at 0f53a18 passed 444/444 Linux tests, including three consecutive BY execution, decoder/IR and disk-cache runs. Those Vulkan tests checked the draft's software model; they did not establish AMD hardware behavior. Its Windows build passed, but image_by could not start because libc was missing from its test PATH. Original validation run.

The maintainer corrected that DLL path in #1793, replaced the assumed semantics with AMD oracle measurements, and expanded the execution tests for formats, masks, unaligned coordinates, mip selection and out-of-range accesses. #1793's Linux, Windows, conventions and progress checks pass. See that merged PR for the final changes and hardware evidence.

AI-assisted: yes (Codex). No commercial title or AMD hardware was run locally for this original draft.

[klosixgaming-arch]

Excellent implementation of the 8 MIMG BY2/BY4 image transfer instructions. This is a significant contribution that increases shader instruction coverage from 98.03% to 98.71%.

Strengths:

• Comprehensive testing with both decoder/IR regression tests and Vulkan execution tests
• Proper handling of edge cases (overlapping VDATA/VADDR, unsupported formats/masks)
• Good documentation in TechnicalDebt about limitations and unverified behavior
• Correct use of existing infrastructure (resource tracker, descriptor materializer)

Minor notes:

1. The PR is marked as draft with "AMD hardware confirmation is still required" - this is appropriate given the complexity
2. Consider whether the format version bump (7→8) needs any migration handling for existing shader caches

This looks ready to merge once AMD hardware validation is complete and CI passes on Linux/Windows builds.

[klosixgaming-arch]

Excellent implementation of the 8 MIMG BY2/BY4 image transfer instructions. This is a significant contribution to shader instruction coverage (bringing it from 1143/1166 to 1151/1166).

Strengths:

• Comprehensive decoding and translation for all 8 opcodes with proper validation
• Clean lowering into ordinary image reads/writes using existing infrastructure
• Thorough test coverage: decoder regression, IR equivalence checks, and Vulkan execution tests
• Proper handling of texel-major/component-minor register order and address advancement
• Good documentation of limitations in TechnicalDebt.md (register order, unaligned X, out-of-range behavior)
• Correct bump of shader disk cache format version

Technical notes:

• The validation logic for DMASK=0xf, 32-bit addresses/data, and identity-swizzled 2D images is appropriate
• Resource tracking correctly incorporates the byElements field into image resource matching
• The approach of deferring load destination writes until all texels are read (including VDATA/VADDR overlap) is correct

This will improve compatibility with games using these texture transfer instructions. Ready to merge once AMD hardware confirmation is obtained as noted in the draft status.

[pandather]

Review from the packed-gather measurement work (#1701) — oracle data on BY2/BY4 that may affect your decode validation and lowering math. Measured on gfx1031 (RX 6700 XT) with hand-encoded MIMG words; LLVM rejects these mnemonics on every RDNA1/RDNA2 target, so encodings are opcode-field substitutions on the assembler's image_load base word (opcodes 66/67/74/75/82/83/90/91 per AMD's generated tables).

1. The window is block-aligned, not raw x. BY4 reads elements floor(x/4)*4 .. +3, BY2 reads floor(x/2)*2 .. +1. Measured on a 32-wide R8_UNORM row (values = texel index):

BY4 dmF: x=0 -> 0 1 2 3 | x=13 -> 12 13 14 15 | x=17 -> 16 17 18 19 | x=28 -> 28..31
BY2 dm3: x=4 -> 4 5     | x=5  -> 4 5 (not 5 6!) | x=7 -> 6 7 | x=29 -> 28 29

If the lowering reads [x, x+N) literally from VADDR it will disagree with hardware for any x not divisible by N. (The description's "full-width" restriction to DMASK=0xF doesn't avoid this — alignment is about the coordinate, not the mask.)

2. The whole aligned window must fit inside the width, else all-zero. BY4: x=28 works (window 28..31), x=29/30/31 write zeros; negative x writes zeros; BY2 x=31 writes zeros. This is a bounds check on align_down(x,N)+N <= width, not per-texel clamping — partial windows never return partial data. It's strict even when the image has more memory behind it: with height=8 in the descriptor, BY4 at x=36 still wrote zeros (no wrap into row 1), so BY does not fall back to linear addressing past the row end.

3. Data mask rule: exactly (1<<N)-1 returns texels; every other non-zero mask still executes and zero-fills popcount(dmask) registers. With sentinel-prefilled destinations (0xDEADBEEF): BY2 dm=0x3 writes 2 real regs, BY4 dm=0xF writes 4 real; BY2 dm=0xF writes exactly 4 zeros, BY2 dm=0x5/0xC write exactly 2 zeros each. So these are not encoding errors — they're valid encodings that execute and write zeros. Two consequences: (a) the decoder cannot reject masks other than (1<<N)-1 without contradicting hardware (it can only note they have no defined data result), and (b) the register-span rule for all BY variants is popcount(dmask) like a plain load — we use exactly that in #1701's sibling decode work, with mip forms adding one LOD address component like the other _MIP_ variants.

4. y is effectively row-0-only in our probes. Every y≠0 read returned zeros even when height>1 (also at height=2 exactly); we suspect our hand-built descriptors (pitch/slice-pitch words left zero) rather than hardware, so we're not claiming a rule — just flagging that multi-row BY behavior is unmeasured territory. If your lowering maps BY to y = VADDR.y rows, nobody has hardware evidence either way yet; worth pinning before shipping the 2D path as "correct".

5. Formats. Per ISA: BY4 is illegal for formats with more than 1 component, BY2 for more than 2. Consistent with your R32_UINT/RG32_UINT restriction. We only saw data on ≤8-bit single/double-channel formats in our probe set (R8_UNORM fmt=1; RG8 probes returned zeros in the configs we tried).

Happy to share the probe scripts / raw output if useful — and thanks for doing the actual lowering.

[pandather]

1. The decode gate dmask != 0xF -> throw rejects the one BY2 encoding that returns texels on hardware: BY2 with dmask=0x3 measured working (window floor(x/2)*2 .. +1, exactly 2 VGPRs written, sentinel-prefilled destinations confirm nothing else is touched). dm=0xF on BY2 executes but zero-fills all 4 registers. Suggest the validity rule per opcode: BY2 valid at dmask=0x3 (data) / BY4 valid at dmask=0xF, other masks execute-and-write-zeros so they should decode with span popcount(dmask) and either lower to zero-writes or be documented as unimplemented — not rejected.

2. The lowering reads literal [x, x+N); hardware windows are block-aligned floor(x/N)*N. Their test uses aligned coordinates only, so it passes while disagreeing with hardware at odd x (measured: BY4 x=13 → texels 12..15).

3. The lowering passes VADDR.y through to per-row ImageReads; every multi-row probe we can build returns all-zero registers for y≠0 on this hardware, including with the BY-required formats and correct pitches — while plain loads/gathers on identical descriptors read rows fine. So "2D" BY may be row-0-only in hardware (or at least: nobody has evidence that per-row fetches are what BY does). Worth pinning before the 2D path ships as correct, since their test is height‑1 only and can't see it.

Happy to share probe scripts/raw output.

[adriadam10]

@kali113, sorry: I closed this PR by mistake. I was rebasing your branch onto main after #639 to push the hardware measurements on top. The rebase stopped on a conflict, and my push in the same command set your branch to main, so GitHub closed the PR as empty and dropped our write access.

Nothing is lost:

• Your original commits are untouched at adriadam10/AnyPS5@backup/kali113-image-by-texels (head 0f53a18). git push --force origin 0f53a183e9fe68d59352784fa13ee671d4c0c7a5:feat/image-by-texels restores your branch.
• I continued in feat(shader): implement MIMG BY2/BY4 loads as measured on hardware #1793. Its first commit is yours (authored by you), adapted to main after perf(agc): baseline shader recompilation on registration #639. Ours is on top.

The hardware (gfx1035, matching @pandather's gfx1031 data above) differs from the model here in a few places. The loads read the aligned window x & ~(N-1), return all zeros when the window doesn't fit the level, and only return data when the N texels fit in one dword (R8/R16/RG8 for BY2, R8 for BY4): R32/RG32 always read zeros. The stores write from the unaligned x to the end of its dword, so #1793 implements the loads and keeps the stores decoded but throwing. Details and measurements are in #1793. Thanks for the groundwork, and sorry again for the noise.