fix(agc): decode no CMASK for single-sample color targets

[winstrex]

What

DecodeColorBuffer decodes a CMASK only for multisampled color targets. A single-sample target with CB_COLOR_INFO.FAST_CLEAR (bit 13) now decodes without one: cmaskAddress stays 0 and CB_COLOR0_CMASK is not read. Multisampled targets are unchanged.

This follows what AMD's two public drivers do on GFX10. The PS5 GPU is GFX10-family.

• Mesa (commit 1b75efe0):
  • ac_surface.c L3031-3036 computes a single-sample CMASK only for gfx_level <= GFX9 ("CMASK -- on GFX10 only for FMASK");
  • ac_descriptors.h L57-58: CMASK_NOAA_COLOR_CLEAR_REG and CMASK_NOAA_COLOR_EXPANDED are "only valid on gfx6-9".
• PAL, AMD's driver (commit c5e80007):
  • Gfx9Cmask::UseCmaskForImage, gfx9MaskRam.cpp L4874-4882 gives a GFX10 image a CMASK only for EQAA or samples > 1;
  • gfx9ColorTargetView.cpp never sets FAST_CLEAR on GFX10 and sets COMPRESSION only with CMASK/FMASK.

Neither driver measures what the color block does with FAST_CLEAR set on a single-sample target. Nothing here is measured on PS5 hardware either.

Why

The Pathless (PPSA01825, own dump, 02.000.000) aborts at its first present:

[videoout] presentation failed: AGC graphics: CMASK codes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 were recorded, which are not a single-sample fast-clear code (0 cleared, 15 expanded)

Its two 3840x2160 display buffers are single-sample, have no DCC and set FAST_CLEAR (CB_COLOR_INFO 0xa828). Their CB_COLOR0_CMASK (0x1010480000, 0x10140d0000) points into memory the title uses as a CPU heap. I measured this with temporary local traces, not committed:

• The range is mapped once and is all zero at the mapping. No other virtual address maps it, and sceKernelEnableDmemAliasing is never called.
• Guest-CPU write faults show who writes the range. First the title's executable writes once per 16 KiB, like an allocator laying out blocks. About 7 s later the title's own libc.prx copies data in.
• At the first draw into the target, the range starts with a PNG signature (89 50 4E 47), and every nibble value 0 to 15 occurs.
• Nothing on the GPU side writes the range before that draw:
  • no recorded GPU write, no driver write, no file read into it;
  • no PM4 packet on any queue carries its address;
  • APS5_NO_FILL_HLE=1 gives the same abort.

So nothing ever fast-clears that CMASK. Reading it as one gives meaning to PNG bytes. Code 0 nibbles would store the clear value over what the title draws, and the other codes abort.

To reach this point, The Pathless needs #4491 (PlayGo chunk ids without a chunk table). Without it, it stops earlier, out of memory.

Tests changed

cmaskTests (#1072, #2470) and one check of multisampleTests (#419) encoded a single-sample CMASK model that was never measured on hardware. They now expect the new decode, and each still checks something:

• A single-sample FAST_CLEAR target decodes without a CMASK. The same registers with 2 samples decode the CMASK.
• The FMASK_SW_MODE / CMASK_PIPE_ALIGNED refusal is checked on the 2-sample target, since a single-sample target no longer reaches it. The mipmapped and misaligned-CMASK refusals were single-sample only and are gone with that path. A missing CMASK address is still refused for multisampled targets in multisampleTests.
• An eliminate pass over a single-sample FAST_CLEAR target leaves its texels and the memory CB_COLOR0_CMASK points at untouched.
• The eliminate and DCC-decompress checks of the CMASK path still run. Their pass now gets the CMASK address set directly, as agc_driver_cmask_fast_clear already does.
• A CB resolve into a single-sample FAST_CLEAR destination now decodes, with no CMASK, instead of throwing.

agc_driver_cmask_fast_clear is unchanged and passes.

TechnicalDebt

docs/dev/debt/libSceAgcDriver.md:

• a new "Single-sample CMASK" entry records that a single-sample FAST_CLEAR is ignored, following Mesa and PAL, and that this is unverified on PS5 hardware;
• "CMASK fast clears" now says the decode no longer reaches that path;
• "Multisampled CMASK" drops its reference to single-sample targets;
• "Multisampled targets in CB passes" says a resolve into a fast-clear destination is accepted: a single-sample destination has no decoded CMASK, and a multisampled destination is already rejected.

Unused code

materializeCmaskClear, CmaskLayout, CmaskClear.comp and TextureDetiler::DispatchCmaskClear are now reached only by the tests: no decoded target has a single-sample CMASK any more. This PR leaves them in place. If you agree with the decode change, I can remove them in a separate PR.

Related

#4041 (PPSA18391, a CMASK the title never initialized) handles codes other than 0 and 15 in the single-sample path. With this change that path is no longer reached from a decoded target. Both PRs edit cmaskTests.

Tested

• Windows 11, RTX 5070 Ti, WinLibs GCC 15.2, Release, on main 2a5eca1, and again after the rebase onto main 1acfa7f (the AGC tests and check_conventions).
  • ctest: 963 of 973 pass.
  • The 10 failures fail the same way on main without this change: agc_driver_flat_loads*, agc_driver_global_loads*, agc_driver_flat_apertures_2/_5, agc_driver_pixel_interlock, agc_driver_wave32_subgroup.
  • agc_driver_graphics fails without the State.cpp change ("a single-sample FAST_CLEAR target decoded a CMASK").
  • tools/check_conventions.py passes.
• The Pathless (PPSA01825) with fix(libScePlayGo): answer chunk ids from the package layout #4491 applied locally:
  • the CMASK abort is gone, and it renders the Annapurna Interactive and Giant Squid logos;
  • it then stops after 33 to 40 s on unrelated errors: a Wwise access violation in one run, storage image access to depth/stencil surface ... is not implemented in another.
• LEGO Horizon Adventures (PPSA14632, 01.004.000, own dump; relinked with --to-intel), the title of feat(agc): render into CMASK fast-clear color targets #1072:
  • Its display buffers have the same register setup as The Pathless's, and its GPU fills their CMASK with 0 every frame.
  • I compared the presented frames with and without this change by frame number (APS5_DUMP_FRAMES with APS5_NO_GPU_DUMP=1, every 5th present). Both show the same loading screen: a black frame with the same spinner, about 1440 to 1475 non-black pixels in the same box, with no trace of earlier frames. Then both present black frames.
  • On my machine it gets no further than that loading screen. Both builds stop after 20 to 75 s at color views of several array slices are unsupported, so I could not compare the intro that feat(agc): render into CMASK fast-clear color targets #1072 checked.
• Full CI on my fork passes after the rebase (run 38294847746: Linux and Windows builds and tests, the relinker, register-context and write-watch jobs on Linux, macOS and Windows). Linux and macOS not run locally.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA01825 (also needs fix(libScePlayGo): answer chunk ids from the package layout #4491)
• AI-assisted: yes

[MarcZX10]

The current conventions check fails with four errors because these four libSceAgcDriver debt entries are being added to the aggregate docs/dev/TechnicalDebt.md. The trusted convention script now requires debt for a library under docs/dev/debt/<library>.md; this component’s file is docs/dev/debt/libSceAgcDriver.md. Could these entries be moved there? That should also clear the reported CI errors.

[MarcZX10]

Thanks, the follow-up commit moves the entries to docs/dev/debt/libSceAgcDriver.md. I rechecked the current head: conventions now passes.

[winstrex]

Thanks! Rebased onto current main: the four entries are now in docs/dev/debt/libSceAgcDriver.md, and the conventions check passes. Full CI on my fork after the rebase is green.

[alfalfa-star]

A data point that may cut against this, from Kena: Bridge of Spirits (PPSA01802, UE4), on a local stack of main + #425 + #3763 + #4031 + #4041 + #4500 + #4586. RADV, RX 9070 XT.

• Kena's 1600x900 R11G11B10 single-sample color targets (half-resolution lighting and post-process passes) have FAST_CLEAR set and a CMASK. The title zero-fills that CMASK itself (a 64-group fill of 0x4000 bytes) before drawing, so it relies on the fast clear.
• With fix(agc): keep the texels of CMASK tiles with codes other than 0 and 15 #4041 alone (code 0 clears the tile, other codes keep the texels), the in-game image has plausible exposure.
• With this PR added, those targets decode no CMASK, the clears are never materialized, and the frames go white and overexposed. That fits the lighting buffers keeping earlier frames' contents.

So at least this title's AGC path fast-clears single-sample targets through CMASK on the PS5, whatever Mesa and PAL do on desktop GFX10. The caveat: Kena's runs vary a lot from run to run, so this is two runs each way, not a clean A/B. I can log the CB_COLOR*_INFO and CMASK registers of those targets if that would help.

[MarcZX10]

This is a concrete counterexample to the assumption that these single-sample targets can ignore CMASK. Could you share the matching CB_COLOR*_INFO and CMASK register values (and the relevant target addresses) from the same Kena scene with #4041 alone and with #4678? If the register state is the same and the result repeats, I would treat the overexposure as a blocker for this decode change; the two-run variability is the main uncertainty left in the current report.

[winstrex]

Thanks for the Kena data point, that settles it: a title that zero-fills its CMASK and relies on the fast clear of single-sample targets rules out ignoring the CMASK.

I checked The Pathless (PPSA01825) on main + #4041 (+ #4491 for PlayGo), without this PR. It gets past the CMASK abort and renders the Annapurna Interactive and Giant Squid logos. The Annapurna frame is pixel-identical to the one with this PR. LEGO Horizon Adventures (PPSA14632) shows the same loading screen as before. Since #4041 covers both titles without dropping the single-sample model, I'll close this PR in favour of that approach.

Details for The Pathless on #4041: #4041 (comment)

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#4603	docs/dev/debt/libSceAgcDriver.md L65-66

[MarcZX10]

Agreed. The Kena result is enough to make ignoring CMASK unsafe, and #4041 has now been checked against The Pathless without this change. Closing this in favor of #4041 is the right direction; thanks for gathering the follow-up title evidence.