fix(agc): color compare emulation: review follow-up to #668

[ThisIsAkill]

Update after review: rebased onto main cea1ef13. The reduction check is dropped (#815 has it). The array layer is now round-half-even (measured on RDNA2 by adriadam10), correcting floor(z + 0.5) from #668. NaN rows were added, and TRUNC_COORD with bilinear is documented. See the reply below for details.

What

A follow-up to #668, which was merged before the review fixes were pushed. It carries the changes that answer the four review points from that PR; nothing else. Two commits on current main:

1. fix(agc): compare NOT_EQUAL unordered and refuse reduction filters and TRUNC_COORD in the color compare emulation
2. test(agc): cover 2D arrays and GREATER in the color compare emulation

How each review point is addressed

1. 2D array. EmulatedColorCompare gains a 2D array case (two layers, a different red table per layer): it covers the layer size query and the floor(z + 0.5) clamp, with layer coordinates below 0, between layers and past the last one.
2. NaN. NOT_EQUAL is now OpFUnordNotEqual (true when the reference or the texel is NaN). I could not verify the hardware's NaN behaviour, so the TechnicalDebt entry says so: it is the IEEE != and as Vulkan defines it for its own depth compare, not confirmed on hardware.
3. Sampler fields. FILTER_MODE (word 0 bits 29 to 30) with a min/max reduction mode throws, and so does TRUNC_COORD (bit 27) with point sampling. The test checks both are rejected and that the default values still run. The TechnicalDebt entry lists both.
4. Other compare functions. The test now runs GREATER as well as LESSEQUAL, which guards the operand order.

Notes

• perf(agc): baseline shader recompilation on registration #639 (still open, and already conflicting with main) touches ResourceMaterializer.cpp, which this change edits by four lines. Whichever lands second rebases; the emulated compare then needs the runtime-mode form you described in the review.
• Nothing here changes behaviour for titles that don't use a compare sampler on a color texture.

Tested

Linux, GCC, NVIDIA RTX 3070, on main at 5895ea72:

• agc_driver_emulated_color_compare passes, 4 runs in a row (the new 2D array, GREATER and rejection cases included).
• The full ctest passes: 359 of 359, with one test (agc_driver_pixel_valid_mask) skipping itself.

I did not run the new cases against the unfixed code in this round.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted:

[ThisIsAkill]

Removed code comments per CONVENTIONS.md; no functional change.

[adriadam10]

Thanks for the follow-up. I measured the four points on RDNA2 (gfx1035, image_sample_c_lz on a 32_FLOAT texture, point sampling) with the HW oracle (tools/hw-oracle/):

• Array layer rounding (blocking). The hardware selects the layer as clamp(roundEven(z), 0, layers - 1), not floor(z + 0.5): with 4 layers, z = 0.5 → 0, 1.5 → 2, 2.5 → 2, 3.5 → 3, 0.49999997 → 0 (in f32, 0.49999997 + 0.5 rounds to 1.0, so floor picks layer 1 there too). SpirvImageEmitter.cpp:936 (from #668) uses Floor(z + 0.5), and the new test pins z = 0.5 → layer 1 (EmulatedColorCompare.cpp:129, Expected at :168), i.e. it locks in the value the hardware disagrees with. GLSLstd450RoundEven (as Vulkan does for its own array layer) and expected values from round-half-even, with rows 0.5, 1.5, 2.5 and 0.49999997 over at least 3 layers, would fix both.
• NOT_EQUAL. Measured: NOT_EQUAL returns 1 when the reference, the texel or both are NaN; the other functions return 0 for NaN; +0 and -0 compare equal. So OpFUnordNotEqual matches. The TechnicalDebt line can say it was measured on RDNA2 instead of "not verified". No test row covers it now (the test never runs NOT_EQUAL, so it passes with OpFOrdNotEqual too): a case on an emulated float format (e.g. 16_16_16_16_FLOAT; R32 float takes the native path) with a NaN texel and a NaN reference under NOT_EQUAL would pin it.
• Rebase. This conflicts with main: #815 already throws for a min/max reduction sampler in emulatedCompareState and adds the matching Reject in this test. Please drop the reduction check, its Reject and the reduction field, and keep the TRUNC_COORD part.
• TRUNC_COORD with bilinear is accepted. On hardware it changes the bilinear weights (e.g. 2.12109 instead of 2.125 at u = 2.4999 texels). That is the same gap as the existing "full float weights" sentence, so either throw for it too or name it in that sentence.

[ThisIsAkill]

Note: the branch is rebased onto current main, as requested, so the commit hashes have changed (now fbbe4789, cfcd3d40, 593ffa19).

@adriadam10 thanks for measuring this on hardware. All four points are addressed, and the branch is rebased onto current main (cea1ef13).

• Array layer rounding. You're right, and the bug is mine: floor(z + 0.5) came in with fix(agc): emulate comparison sampling of color textures in the shader #668, so this corrects behaviour that's already on main. The layer is now clamp(RoundEven(z), 0, layers - 1) (GLSL.std.450 RoundEven, as Vulkan uses for its own array layer). The test now uses 4 layers whose neighbours are far apart in value, with layer coordinates 0.49999997, 0.5, 1.5, 2.5, 3.5, and values below 0 and past the last layer. Expected values use round-half-even. With the old floor(z + 0.5) put back, the test fails at z = 0.49999997.
• NOT_EQUAL and NaN. OpFUnordNotEqual stays, and TechnicalDebt now says it was measured by you on RDNA2 (gfx1035): NOT_EQUAL is true for NaN and the other functions are false. There are two new rows on a 16_16_16_16_FLOAT texture (emulated path), NOT_EQUAL and LESSEQUAL, with NaN, +0 and −0 in both the reference and the texel. With OpFOrdNotEqual put back, the NOT_EQUAL row fails.
• Rebase. Dropped my reduction check, its Reject and the reduction field, since feat(agc): implement min and max sampler reduction modes #815 covers them. Kept the TRUNC_COORD refusal for point sampling.
• TRUNC_COORD with bilinear. Documented in the full-float-weights sentence ("TRUNC_COORD, which changes those weights on hardware, is accepted under bilinear filtering without that change"), rather than throwing.

Tests: agc_driver_emulated_color_compare passes on Linux with GCC on an RTX 3070. The full ctest (default build) passes, 399 of 399.

[adriadam10]

Ready to merge from my side, but the PR description fails the conventions check:

• [pr-template] Fill in the pull request template: unchecked: AI-assisted:
• [pr-template] Fill in the pull request template: answer AI-assisted with yes or no

Please fill those template lines in (only you can answer AI-assisted); it merges once they pass.

[oneandonlydean]

Conflicts with main in SpirvImageEmitter.cpp, EmulatedColorCompare.cpp and TechnicalDebt; please rebase. Main still has Floor(z + 0.5) for the layer and OpFOrdNotEqual (SpirvImageEmitter.cpp:1307, :1343), so both fixes are still needed.

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 1341	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 1368	docs/dev/TechnicalDebt.md
PR 1384	docs/dev/TechnicalDebt.md
PR 1395	docs/dev/TechnicalDebt.md
PR 1601	docs/dev/TechnicalDebt.md
PR 1835	docs/dev/TechnicalDebt.md
PR 2780	docs/dev/TechnicalDebt.md
PR 3986	docs/dev/TechnicalDebt.md
PR 5021	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp docs/dev/TechnicalDebt.md
PR 5371	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 5958	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6039	docs/dev/TechnicalDebt.md
PR 6218	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6232	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6280	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6312	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6412	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6440	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6454	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6488	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6674	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 6758	docs/dev/TechnicalDebt.md
PR 6799	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6815	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6947	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 6977	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 7085	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7223	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7296	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 7332	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7381	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 7395	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7550	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp