fix(agc): skip unused pixel stages with retained export formats

[moong15]

What

Skip a bound pixel program when it has no enabled consumers or execution requirements, even if SPI_SHADER_Z_FORMAT retains a previous export format. Stellar Blade (PPSA13197 v01.141) issues depth-only draws in this state during loading. Main materializes the retained shader against the next path's unrelated userdata, producing an oversized buffer descriptor and a GPU wait/watchdog failure. In matched runs this is intermittent on main rather than a deterministic boot blocker; see the runtime results below.

The captured sequence clears CB_TARGET_MASK, CB_SHADER_MASK, and DB_SHADER_CONTROL, while retaining Z export format 1 and the preceding pixel program. Full shader-code comparison matched live memory, and the userdata matched the direct command writes. That evidence points to executing an unused stage, rather than stale shader code or corrupted register writes.

This change:

• Removes Z export format as an independent execution requirement.
• Adds COVERAGE_TO_MASK_ENABLE to the existing export/execution mask.
• Retains the pixel stage whenever DB_ALPHA_TO_MASK is enabled. This is conservative even with ALPHA_TO_MASK_DISABLE, because the current Vulkan state decoder does not apply that override.
• Preserves depth/stencil/sample-mask exports, kill, coverage, and the existing side-effect execution flags. Unsupported export behavior still follows the existing rejection paths.

The semantics come from AMD PAL, not a new console hardware measurement: DbShaderControl, including its explanation of no-consumer PS suppression, SpiShaderZFormat, and GFX10 register definitions. The inferred behavior and alpha-to-coverage limitation are recorded in technical debt.

Tested

Controlled comparison on main 2ba3147ac804a2e6e82d558a08f752d6627d4a74 (PR head 660d2e76a), Windows 11 Pro 10.0.26200, AMD Radeon RX 9070 XT, Windows driver 32.0.32015.2008, GCC 15.2.0 UCRT, Release, SPIRV-Tools enabled. A separate worktree contains only this PR's changes; third-party inputs use the existing pinned checkouts. No unrelated local source changes or game settings participate in these tests.

Both runs use identical final regression sources. The before run builds Graphics/src/State.cpp exactly from main; the after run recompiles it with this fix.

Suite	Main predicate + new regressions	This PR
agc_driver_graphics	Fails: retained Z format incorrectly keeps the pixel program active	Pass
agc_driver_prepared_graphics	Fails: a valid retained 20-word program is still consumed instead of being substituted with no pixel program	Pass
agc_driver_draw_cache_reuse	Pass	Pass
agc_driver_draw_cache_reuse_off	Pass	Pass
agc_driver_register_key	Pass	Pass

Regression coverage includes formats 0/1/2/3/9; every export/execution bit; valid depth, stencil-reference and sample-mask export combinations; missing execution state; disjoint and high color masks; alpha-to-coverage and its override; and fixed-function depth/stencil operations and clear modes compared with an explicitly unbound pixel program. Preparation uses a nonzero retained address, the captured valid 20-word user SGPR count with 30 unrelated userdata words, a separate poisoned 63-word count, poisoned fragment input/export state, wave32/wave64, linked fragment roles, and draw-time variant reuse. Re-enabling color, EXEC_ON_NOOP, or alpha-to-coverage restores the real program. The cache regression switches four states over 21 draws and checks four entries and 13 hits, both with and without GPL. Both cache CTest variants now set APS5_VERIFY_DRAW_RECIPE=1 and APS5_VERIFY_REGISTER_KEYS=1 themselves, and the test asserts that fresh-decode verification actually ran with no mismatches, so standard CTest covers it without extra environment.

Reproduce with ANYPS5_GPU="AMD Radeon RX 9070 XT", ANYPS5_REQUIRE_VULKAN=1 and ANYPS5_NO_SHADER_CACHE=1:

cmake --build build --target agc_driver_graphics_tests agc_driver_prepared_graphics_tests agc_driver_draw_cache_reuse_tests agc_driver_register_key_tests
ctest --test-dir build -R '^agc_driver_(graphics|prepared_graphics|draw_cache_reuse(_off)?|register_key)$' --output-on-failure

For the failing-before comparison, restore only core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp from the base revision in a disposable checkout, rebuild, and run the same command. Restore the PR version afterward. All five focused suites pass with the fix; each finishes below 30 seconds. tools/check_conventions.py --base origin/main and whitespace checks pass. Sanitized test logs: gist.

October 14: matched game runs, main against this PR. Clean main 2ba3147 against this PR's head 660d2e7, Stellar Blade PPSA13197 v01.141, same converted executable. The deployed libraries are identical except libSceAgcDriver.prx (main 76d86473…, PR f1dd7aac…), checked by SHA-256 on every run. Each run cleared inherited APS5_*/ANYPS5_* variables and set only ANYPS5_GPU="AMD Radeon RX 9070 XT", APS5_HOST_IMPORT_MIB=8192 and ANYPS5_SHADER_CACHE_DIR (the run's private seed copy). The runner stopped a game whose private commit exceeded 42,000 MiB or when free memory fell below 3,000 MiB; that never happened. It also set APS5_COMMIT_LIMIT_MIB/APS5_MIN_FREE_MIB, which AnyPS5 does not read. No tracing, image capture, runner timeout or other local patches were used. Each run started from a private copy of a shared shader-cache seed. Input was manual: only the gift dialog was claimed. The menu was confirmed by eye at the screen, then the game was closed by hand. All main runs of a seed ran before its PR runs; they did not alternate. No builds or tests ran concurrently.

Seed 1 (6,818 files, 2,602,396,833 bytes), built by one cold main run, excluded:

Run	main	this PR
1	FATAL 320 s: presentation, guest sampler descriptor uses a secondary LOD bias which is not implemented	menu by 530 s, closed at 684 s
2	FATAL 320 s: presentation, buffer descriptor uses an unsupported type	engine watchdog, fault at 321 s
3	engine watchdog, fault at 313 s	menu by 503 s, closed at 630 s

Seed 2 (10,795 files, 5,612,809,177 bytes), the final cache of PR seed-1 run 1:

Run	main	this PR
1	FATAL 117 s: presentation, buffer descriptor uses an unsupported type	menu by 153 s, closed at 255 s
2	menu by 184 s, closed at 322 s	menu by 206 s, closed at 278 s
3	menu by 319 s, closed at 452 s	menu by 210 s, closed at 223 s

Main reached the menu in 2 of 6 runs; this PR in 5 of 6. Three of six main runs ended in a host exception on a malformed descriptor during presentation, followed by abort(); no PR run did. The failure is intermittent on main, not a deterministic boot blocker.

The engine watchdog is Unreal's fixed 120 s GameThread timed out waiting for RenderThread, followed by a deliberate guest access violation. It hit both builds, and only on seed 1. Seed 1 ends where the cold run stopped, so warm runs on it still compiled about 1,400–1,700 shaders late in loading. Runs on seed 2 added at most 22 cache files, and none hit the watchdog. This change does not address that stall. WAIT_REG_MEM timeouts (2–5 per run) occurred on both builds with seed 1, including PR runs that reached the menu, and on neither with seed 2. Times are from process creation; "menu by" is when the menu was reported.

Every main-only failure was a malformed descriptor, the failure class described in What: on main the retained pixel program reads the next draw's unrelated userdata as its resources. None occurred with this change. The native tests show the mechanism directly; the game runs show the matching outcome, though they did not trace each exception back to a specific draw. No gameplay, cold-cache reliability, other-title, Linux, console or other-GPU claim is made. The original game libraries were restored and hash-checked afterward. Per-run exits, peak commit, wait timeouts, cache totals and the only extracted log lines (paths and addresses removed) are in runtime-runs.md.

Earlier title runs (October 13, other local patches): with APS5_FLAT_3D_COLOR_TARGET=1, APS5_CONTINUE_AFTER_FLIP_FAILURE=1 and 16 prebuild threads, three fresh launches reached the menu. That is not a clean-main comparison; the matched runs above supersede it.

Related open #5304 and #5521 address unused stencil export and alpha-to-coverage output support respectively; neither implements this skip predicate, and neither is a dependency.

Checklist

• Based on current main; no other open PR implements the same fix
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw; no silent stubs added
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source — none added
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: removes an intermittent Stellar Blade PPSA13197 v01.141 loading crash on main (no Game bring-up discussion found)
• GPUs tested: AMD Radeon RX 9070 XT, Windows driver 32.0.32015.2008, Windows 11 Pro 10.0.26200
• AI-assisted: yes (OpenAI Codex, Claude Code)

🤖 Generated with Claude Code

[tibzejoker]

Runs for this PR at head 660d2e76a. Windows 11, MinGW-w64 GCC 15.2.0 Release, two GPUs in one machine: Intel UHD Graphics 770 (driver 32.0.101.6129, picked with ANYPS5_GPU=Intel) and RTX 4090 (NVIDIA 616.92).

• The tests built from this PR's changed test files (agc_compute_descriptor_layout_flags, agc_descriptor_pool_reset_failure, agc_driver_address_space_teardown, agc_driver_covered_centroid_state, agc_driver_draw_cache_reuse, agc_driver_draw_cache_reuse_off, agc_driver_graphics, agc_driver_prepared_graphics, agc_sparse_buffer_holes) pass on both GPUs.
• ctest -R "agc_driver|agc_masked|agc_shader" -j4: no failures on either GPU. Intel: 395 tests, 28 skipped. NVIDIA: 395 tests, 1 skipped (agc_driver_stencil_ref_export). The skips are the same as on main with each GPU. On the UHD 770 they are features it doesn't support: f64, 64-bit and float atomics, mesh shaders, fp16 interpolation and dot, shader clock.

No difference between the two GPUs.

AI-assisted (Claude Code).

[simonbronk]

AMD and NVIDIA runs for this PR at head 660d2e76a. Windows 11 Pro 26200, WinLibs GCC 15.2.0 (MinGW-w64), Release, one machine with two GPUs, each picked with ANYPS5_GPU:

• NVIDIA GeForce RTX 5070, driver 617.42 (32.0.16.1742)
• AMD Radeon(TM) Graphics, Ryzen integrated, gfx1036 (1002:13c0, RDNA 2), driver 32.0.21036.18

ctest -R "agc|shader|video_out" -j 8 (511 tests, including the ones this PR adds or changes):

• RTX 5070: all 511 pass.
• AMD gfx1036: all pass except agc_driver_lds_mixed_width_atomics, agc_driver_pixel_interlock and agc_driver_layered_volume_vertex_no_gpl. These three fail the same way on current main (872f52320) on this GPU and pass there on the RTX 5070; they are the three of agc_* tests: 3 of 483 fail on AMD Radeon RX 6400, pass on NVIDIA RTX 4080 SUPER #5271, so not caused by this PR.

No title run. AI-assisted: yes (Claude Code ran the builds and tests).

[p3n1cillin]

A second title this fixes: Stray (PPSA02100, Unreal Engine 4), Windows 11, RTX 5070, MinGW GCC 15, Release, driven with Enter presses (Cross) and run without a display.

Build	Result
main 4c1de0c0c	stops while the first level loads with buffer descriptor uses an unsupported type
main + this PR + #5897	2 of 2 runs reach gameplay: the opening scene with the cats renders, and there's no stop in 250 s and 300 s

The main result matches all 5 earlier runs on older main. The descriptor is a 2D or 3D T# (…91b00f2e / …a1b00fac) in buffer slot 2 of 3 of a pixel shader. With the buffer bound empty locally, the next stop was a 3D T# in a 2D image slot of the same shader, as @nikowolf123 predicted in #5879. Neither stop appears with this PR, which fits a stale pixel program being skipped on a depth-only draw.

AI-assisted: yes (Claude Code)

[ThatsLeo]

Stray (PPSA02101, the EU version of @p3n1cillin's #5879) on Intel, as @nikowolf123 suggested in #5894. Fedora 44, Intel Arc (Meteor Lake iGPU, Mesa 26.2.3 ANV), Release.

Setup. main e154f5658 with these on top:

• feat(recompiler): run single-wave wave64 compute programs at one lane per invocation on 32-lane hosts #5405, plus a local change that lets its spanned waves run at any host subgroup size;
• perf(shader): keep only evaluable branch conditions in resource plans #5181, perf(agc): keep a smaller staging pool on system-memory heaps #5893's staging pool budget, and a local cut of the storage image cache to a sixteenth of the heap;
• this PR at 75083acb3, and fix(agc): name the shader and V# when a draw is refused for a buffer descriptor type #5897.

The extra commits are needed because Stray's post-processing compute programs don't compile in time on ANV (#4977, #5405). Each run holds Enter (Cross) at the calibration screen and goes through the menu into the opening cinematic. It runs in a 12 GiB + 4 GiB swap cgroup (two of the runs with this PR: 13 GiB, no swap) at 0.1 to 0.5 FPS, and I took a screenshot every 30 s.

Build	Runs	buffer descriptor uses an unsupported type
without this PR (an earlier main 66f89476, before #5897)	3	stopped in all 3, at frames 164 to 175
without this PR (this stack with the PR reverted)	2	stopped in 1, at frame 175; the other ended its 420 s at frame 215
with this PR	8	none: up to 278 frames and 20 minutes, the cinematic running

The table counts only runs that left the main menu. With the PR, five runs went past frame 225.

I lost the log of the stopped run on the current stack, so I don't have #5897's line or the context registers for it. In the earlier ones the descriptor was the same T# (20722c00 c3800000 001dc13f a1b00fac) in buffer slot 0 of a pixel shader.

AI-assisted: yes

[oneandonlydean]

Rebased onto main myself: the draw-cache test conflicted with #494, so RetainedPixelCacheTransitions now starts from Draws(32.0f, {}) instead of DrawThrice() minus its three draws. Merging once the build is green.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#6015	docs/dev/debt/libSceAgcDriver.md L48