fix(shader): return 0 for 8-bit UNORM color comparison

[EagleBearerYS]

What

Planet of Lana (PPSA19450) aborts on main before its first frame is presented:

[videoout] presentation failed: comparison sampling of a color texture is implemented only with equal point or bilinear minification and magnification filters
terminate called after throwing an instance of 'std::runtime_error'
FATAL: abort() (SIGABRT)

The shader does an LOD-zero comparison (EmulatedCompare flags 0) on a 1x1 8_8_8_8_UNORM cube with one level, through an anisotropic bilinear S#: MAX_ANISO_RATIO 3, XY_MAG_FILTER = XY_MIN_FILTER = 3, DEPTH_COMPARE_FUNC 6 (GREATER_EQUAL), DISABLE_CUBE_WRAP 0. Descriptors as printed by a local diagnostic:

S#=00006692 00fff000 06f00000 00000000  T#=021d9530 03800000 00000000 b0500fac 00000005 00700000 00000000 00000000  flags=0 dim=Dim2DArray

In the review of #3438, adriadam10 measured on the hardware oracle (gfx1035) that image_sample_c on an 8-bit sRGB "(and 8-bit UNORM)" T# returns 0 for every compare function over references -0.5..1.5 and texel codes 0..255. #3438 implemented that for sRGB and left 8-bit UNORM comparing, because the existing tests pinned a real compare. TechnicalDebt has recorded the difference since then.

colorCompareReference now returns ReferenceZero for Format8UNorm, Format8_8UNorm and Format8_8_8_8UNorm, as it does for 8-bit sRGB. In emulatedCompareState, formats with that constant result skip two refusals:

• the bilinear cube without DISABLE_CUBE_WRAP, already skipped for 8-bit sRGB since fix(shader): return 0 for an 8-bit sRGB bilinear cube comparison #3540;
• the check for equal point or bilinear filters.

Every texel comparison of these formats is 0, so neither the selected cube face nor any filter weighting can change the result. All other formats keep both refusals. The addressing, border-table, reduction and unnormalized-coordinate checks are unchanged for every format. The anisotropic S# was not measured, and TechnicalDebt says so.

Changes to the test, agc_driver_emulated_color_compare:

• Moved: the rows that check real comparisons now use 16_16_UNORM instead of 8_8_8_8_UNORM. Red is written as r8 * 257, so the expected values are unchanged. This covers point, bilinear, wrap, border, offsets, channels, the cube with DISABLE_CUBE_WRAP, and the bilinear-cube and anisotropic refusals.
• New: rows that expect 0:
  • a bilinear 8_8_8_8_UNORM cube without DISABLE_CUBE_WRAP, for all 8 compare functions;
  • the title's anisotropic 8_8_8_8_UNORM cube with GREATER_EQUAL;
  • 2D 8_8_8_8_UNORM for all 8 compare functions, with point, bilinear and anisotropic bilinear filters;
  • 8_UNORM and 8_8_UNORM with LESS_EQUAL and the same three filters.

Both color-comparison entries in TechnicalDebt now list the 8-bit UNORM formats.

No open pull request changes these checks. #907 also edits this test, and its real-compare rows use 8_8_8_8_UNORM. If it lands after this change, those rows need a format that compares, such as the 16_16_UNORM used here.

Tested

Windows 11 Home (10.0.26300), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release, NVIDIA GeForce RTX 3070 Laptop GPU.

• Test without the fix: with main's ResourceMaterializer.cpp, agc_driver_emulated_color_compare fails at the first new row: bilinear cube comparison requires DISABLE_CUBE_WRAP to avoid seamless face filtering.
• Test with the fix: agc_driver_emulated_color_compare and agc_driver_emulated_color_compare_srgb_decode pass 3 runs in a row on main d371ed3. agc_driver_emulated_color_compare passes again on main d146ec1, in a build that also had feat(pad): hold the sticks from the controller command file #4032's unrelated pad change.
• Title, main without the fix: the stop above, on a11c8f7 and d371ed3.
• Title with the fix: Planet of Lana (PPSA19450, 01.010.000) presents its first frame and reaches the main menu, the save-slot screen, New Game, the opening sequence and gameplay in the first village. A save written there is listed on the next start and loads with Continue.
• A separate stop: in 2 of 5 runs with the fix, the game later exited silently while loading that save, with exit code 0x20474343 (an uncaught host C++ exception). It is still being investigated.
• python tools/check_conventions.py --base origin/main passes.

The full suite was not run. Not tested on Linux, macOS or AMD GPUs.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt (no unimplemented paths or stubs added)
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source (no third-party code added)
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#4031	docs/dev/debt/shader.md L5-6
#4842	docs/dev/debt/shader.md L5-6