fix(agc): leave the depth surface out when no enabled test has its plane

[oneandonlydean]

What

DecodeState binds the depth surface when DB_DEPTH_CONTROL enables a depth, stencil or depth bounds test and either plane is valid. It doesn't ask whether the enabled test has its own plane. A stencil test beside a depth-only surface (DB_STENCIL_INFO format invalid, depth test off) tests nothing, but the surface was attached anyway, and the render extent and the scissor were clamped to it.

A title we test with leaves a 960x540 depth surface bound after a half-resolution pass and then draws into its 1920x1080 target with DB_DEPTH_CONTROL 0x007007b1 (stencil test on, always pass; depth test off), DB_Z_INFO 0x80000183 and DB_STENCIL_INFO 0x20000180. The draw's viewport is 1920x1080, but its render extent became 960x540, so a full-screen transition covered only the top-left quarter of the frame.

A depth or stencil test whose plane register is present with an invalid format no longer counts towards binding the surface. decodeDepth already turns such a test off (#453); the surface was bound for it all the same. When no enabled test is left, the draw renders without a depth attachment, as it already does when both planes are invalid (#453, #2179). DrawRejection follows the same rule. A test with a valid plane binds the surface as before, and so does a clear.

Not changed:

• A draw whose depth test has its plane still clamps the render extent to a smaller depth surface, also when the test always passes and writes nothing.
• A depth bounds test still binds the surface when the stencil plane is valid, and still throws depth bounds without a depth plane when the depth plane is invalid.
• That a test without its plane passes is the existing rule of fix(libSceAgcDriver): skip depth and stencil tests whose plane is absent #453 and fix(agc): pass the depth bounds test when both depth planes are absent #2179. It was not checked on hardware here.

Tested

Linux (CachyOS, kernel 7.2.3, GCC 16.2.1, Release with CI's configure flags and ANYPS5_ENABLE_SPIRV_TOOLS=ON), RTX 4090, NVIDIA 610.57.04. One commit, c1287d7, on main a11c8f7.

• agc_driver_graphics has three new cases: an always-pass stencil test and a failing stencil test, each without a stencil plane beside a valid 32x2 depth plane the draw doesn't test, decode without a depth surface, keep the color target's extent and are not rejected; a depth test on that plane still binds it and clamps to 32x2. With only the test change on main a11c8f7 the test fails with a stencil test without a stencil plane bound the depth plane it does not test. With the change it passes.
• The mirrored case (a depth test without a depth plane beside a valid stencil plane the draw doesn't test) was checked the same way with a local test that is not in the commit: no depth surface, the color target's extent, not rejected.
• Full ctest (--parallel 4, ANYPS5_REQUIRE_VULKAN=1), 838 tests, on main a11c8f7 and on this branch: 833 pass, 4 skip (video_out_flip_lifetime, _reopen, _controls, _overlay), 1 fails on both. The failure is agc_driver_emulated_color_compare (linux CI + NVIDIA: agc_driver_emulated_color_compare fails on main since #3438 (8-bit sRGB compare refused on the decode path) #3526).
• The AGC, shader and write-tracking tests (ctest -R "agc|shader|write", 391 tests) on lavapipe (Mesa 26.2.3, LLVM 22.1.8): the same two fail on main a11c8f7 and on this branch (agc_driver_emulated_color_compare, agc_driver_image_plane_address_on_line_implicit_lod), the same 56 skip.
• python3 tools/check_conventions.py --base origin/main passes.
• A second title we test with (a 2D title), one scripted 220 s run on this commit: the menu at 60.0 fps and gameplay at 59.9 fps, all seven frame checks pass, 0 skipped draws, 0 fatal lines, 0 kernel Xid lines.
• The first title above, on main 2220eff with feat(agc): depth planes sampled as 2D arrays, cubes and integer views #425, fix(libSceAgcDriver): retire a depth surface once another writer covers its memory #454, perf(agc): keep unchanged guest layers of a depth plane copy #2490 and perf(agc): skip the depth plane copy when no slice changed #2491 (it needs them to get past its intro) and the same change: without the change every frame of the transition is drawn in the top-left quarter; with it the transition fills the frame in two runs of the scripted sequence (boot, title, menu, first level; no frame of either run has the quarter picture). The frames before and after the transition are the same by eye. In another level of the same title, one run each: 2.65 fps without the change, 2.55 fps with it. It was not run again on main a11c8f7.

Not tested: Windows, AMD, Intel.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes (Claude Code)

[adriadam10]

Checked on current main: build and ctest of libSceAgcDriver (390 tests) green, and agc_driver_graphics fails without the State.cpp change and passes with it. DecodeState and DrawRejection use the same rule, and a clear or a test with a valid plane still binds the surface. Merging.