Skip to content

fix(shader): kill demoted quad-mask pixels when the pixel program returns - #6255

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
LeandroLP23:fix/agc-kill-demoted-quad-pixels
Oct 18, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
LeandroLP23:fix/agc-kill-demoted-quad-pixels

Conversation

@LeandroLP23

@LeandroLP23 LeandroLP23 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What

agc_driver_quad_pixel_mask, added with #6015, fails on AMD Windows: implicit-LOD sample with quad pixel mask 2: pixel (0, 0) channel 2 is 2, expected 0.

A pixel that PA_SC_AA_MASK leaves uncovered is demoted to a helper invocation, so the covered pixels keep their derivatives. The test's sampled program exports under the restored EXEC mask, which clears the uncovered lane, so the recompiled shader never writes that invocation's colour output. On the Radeon 860M the driver still writes something for those demoted invocations. Every uncovered pixel gets undefined colour, not the sampled texel (60df42ff). The value depends only on x, the same in every run:

covered:    x=1,3,5,7   60df42ff
uncovered:  row 0: 00000200 . 00000800 . 00001000 . 00001000 .
            row 1: 00000200 00000400 00000800 00000800 00001000 00001000 00001000 00001000

The Vulkan spec says a helper invocation's outputs are discarded, so this looks like driver behaviour. This PR emits one OpKill for the uncovered invocations at the program's return. By then every derivative has been taken, so the covered pixels keep their LOD, and the demoted pixels are discarded on every driver. Without demote, the uncovered invocations are already killed at entry and the new branch is never taken.

The sample masks entry in docs/dev/debt/libSceAgcDriver.md now mentions the kill and the driver behaviour.

Tested

Windows 11 (26300), Radeon 860M (RDNA 3.5, 1002:1114), driver 32.0.13058.8001, WinLibs GCC 15.2.0, Release, on main d92a8dad3.

  • This PR: agc_driver_quad_pixel_mask, agc_driver_sample_mask_draw, agc_driver_sample_mask_depth_stencil, agc_driver_sample_mask_multisample, agc_shader_quad_mask_no_demote and request_serializer pass, --repeat until-fail:3. The quad pixel mask test reports quad pixel mask tests passed (demote to helper invocation).
  • Negative control: agc_driver_quad_pixel_mask on main d92a8dad3 fails 3 of 3, with GPL on and off.
  • Full ctest -R '^agc_' -j 1 from an empty tests/shader_cache: one failure, agc_driver_layered_volume_vertex_no_gpl. That's the AMD driver cache inversion of AMD Windows: identical AGC executables invert GPL pass/fail results after relocation #4768, which fails in every full run on main on this machine.
  • tools/check_conventions.py --base origin/main passes; git diff --check is clean.

Fork CI (run): Linux 1229 of 1230 and Windows 1252 of 1253, the one failure in each being the fork-only conventions_ci_gate. The first Windows attempt also timed out in agc_driver_readonly_resummarize, a write-tracking test with no quad mask that passes in 0.02 s on Linux and locally; it passed on the job rerun.

Not tested: any title. @savvaniss ran the quad-mask and sample-mask tests on an RX 9070 XT (RDNA 4), where this fixes agc_driver_quad_pixel_mask, and on an Intel UHD 770, where the base already passes and nothing changes. @jotitadev ran it on NVIDIA (RTX 3070, Linux).

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: GTA V (PPSA04263), the title feat(agc): emulate PA_SC_AA_MASK sample masks #6015 added sample masks for, on AMD Windows, where pixels a quad mask leaves uncovered would otherwise get undefined colour (not run here)
  • GPUs tested: AMD Radeon 860M (RDNA 3.5), driver 32.0.13058.8001, Windows 11 26300
  • AI-assisted: yes (Claude Code)

…urns

A pixel that PA_SC_AA_MASK leaves uncovered is demoted to a helper
invocation so the covered pixels keep their derivatives. When the guest
exports under an EXEC mask that clears that lane, the colour output is
never written. On the Radeon 860M (Windows driver 32.0.13058.8001) such
a demoted invocation still writes undefined colour to the target, so
agc_driver_quad_pixel_mask fails at its first implicit-LOD sample under a
partial mask (2). Killing the uncovered invocations at the return, after every
derivative is taken, discards them on every driver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added shader core/shader docs labels Oct 15, 2026
@LeandroLP23
LeandroLP23 marked this pull request as ready for review October 15, 2026 09:51
This was referenced Oct 15, 2026
@github-actions github-actions Bot mentioned this pull request Oct 15, 2026
11 tasks
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 2434 docs/dev/debt/libSceAgcDriver.md
PR 4072 core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp
PR 4448 docs/dev/debt/libSceAgcDriver.md
PR 4650 docs/dev/debt/libSceAgcDriver.md
PR 5021 core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp
PR 5058 docs/dev/debt/libSceAgcDriver.md
PR 5499 docs/dev/debt/libSceAgcDriver.md
PR 5925 docs/dev/debt/libSceAgcDriver.md
PR 6194 docs/dev/debt/libSceAgcDriver.md
PR 6215 docs/dev/debt/libSceAgcDriver.md
PR 6223 docs/dev/debt/libSceAgcDriver.md
PR 6234 docs/dev/debt/libSceAgcDriver.md
PR 6273 docs/dev/debt/libSceAgcDriver.md
PR 6280 docs/dev/debt/libSceAgcDriver.md
PR 6300 docs/dev/debt/libSceAgcDriver.md
PR 6412 docs/dev/debt/libSceAgcDriver.md
PR 6472 docs/dev/debt/libSceAgcDriver.md
PR 6481 docs/dev/debt/libSceAgcDriver.md
PR 6529 docs/dev/debt/libSceAgcDriver.md
PR 6548 docs/dev/debt/libSceAgcDriver.md
PR 6549 docs/dev/debt/libSceAgcDriver.md
PR 6557 docs/dev/debt/libSceAgcDriver.md
PR 6563 docs/dev/debt/libSceAgcDriver.md
PR 6727 docs/dev/debt/libSceAgcDriver.md
PR 6786 docs/dev/debt/libSceAgcDriver.md
PR 6808 docs/dev/debt/libSceAgcDriver.md
PR 6867 docs/dev/debt/libSceAgcDriver.md
PR 6868 docs/dev/debt/libSceAgcDriver.md
PR 6941 docs/dev/debt/libSceAgcDriver.md
PR 6946 docs/dev/debt/libSceAgcDriver.md
PR 6947 core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp
docs/dev/debt/libSceAgcDriver.md
PR 6977 docs/dev/debt/libSceAgcDriver.md
PR 7008 docs/dev/debt/libSceAgcDriver.md
PR 7256 docs/dev/debt/libSceAgcDriver.md
PR 7258 docs/dev/debt/libSceAgcDriver.md
PR 7332 core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp
docs/dev/debt/libSceAgcDriver.md
PR 7390 docs/dev/debt/libSceAgcDriver.md
PR 7429 core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp
PR 7642 core/shader/recompiler/SpirvBackend/src/SpirvFlowEmitter.cpp
docs/dev/debt/libSceAgcDriver.md

@jotitadev jotitadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NVIDIA run, for the "Not tested: NVIDIA" line.

Ubuntu 26.04 (kernel 7.0.0-38), NVIDIA GeForce RTX 3070, driver 595.99.02, GCC 15.2, Release, BUILD_TESTING=ON, ANYPS5_GPU set to the 3070 (the log shows Selected GPU name=NVIDIA GeForce RTX 3070 vendor=0x10de). Head 79d72b683 merged onto main 44df21632, no conflicts.

  • agc_driver_quad_pixel_mask passes on unmodified main and with this PR, 3 of 3 runs with the PR. Both report quad pixel mask tests passed (demote to helper invocation), so the demote path runs. This driver does not show the Radeon 860M behaviour: the demoted pixels already come out right without the kill. The test cannot act as a negative control on this GPU.
  • With the PR, the new OpKill causes no regression here. agc_driver_sample_mask_draw, agc_driver_sample_mask_depth_stencil, agc_driver_sample_mask_multisample and agc_shader_quad_mask_no_demote pass in the full run.
  • ctest -R "^agc" -E "agc_display_buffer_dcc|agc_driver_flip|agc_driver_lds_atomics64|display" -j1: 592 of 593 pass, with the same 3 skips as main. The failure is agc_driver_store_over_untracked, which also fails on unmodified main on this machine: 9 of 10 runs with other builds running, all with the result did not reach a block the CPU left alone. It is the same test as #6347 and #6383, which report it on Windows. On an idle machine, main passes 593 of 593.

Not run: Intel, any title.

GPUs tested: NVIDIA GeForce RTX 3070, driver 595.99.02, Vulkan 1.4.329, Ubuntu 26.04 (kernel 7.0.0-38)

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

@PYTHONRETARDE

Copy link
Copy Markdown
Contributor

RDNA 3 data point: this fixes agc_driver_quad_pixel_mask on an RX 7800 XT too.

RX 7800 XT (1002:747e), driver 32.0.31041.1004, Windows 11 Pro 26200, WinLibs GCC 15.2.0 r7, Release. Baseline main 03c4cb5e7 + #6452 (needed to build on Windows); test build adds this PR's head 79d72b683. %LOCALAPPDATA%\AMD\VkCache emptied before each run (#4768).

Test Baseline With this PR
agc_driver_quad_pixel_mask 5/5 fail: implicit-LOD sample with quad pixel mask 2: pixel (0, 0) channel 2 is 2, expected 0 5/5 pass
agc_driver_sample_mask_depth_stencil, agc_driver_pixel_helper_lanes — 5/5 pass

Full CTest with the PR: 1294 of 1298 pass. None of the four failures involve quad masks: agc_driver_layered_volume_draw and _vertex (#4768 cache swap; 3/3 pass alone), agc_driver_depth_surface_from_storage (fails 1 in 5 on the baseline too) and guest_kqueue (#6173).

AI-assisted: yes (Claude Code).

@savvaniss

Copy link
Copy Markdown
Contributor

RDNA 4 and Intel data points: this fixes agc_driver_quad_pixel_mask on an RX 9070 XT and changes nothing on an Intel UHD 770, where the base already passes.

Windows 11 Pro 26300, WinLibs GCC 15.2.0 r7, Release, every execution test built. Base: main a7f108030 + #6452 (4fe92c9c2, needed to build on Windows); this head 79d72b683 merged on top. ANYPS5_NO_SHADER_CACHE=1; %LOCALAPPDATA%\AMD\VkCache emptied before each AMD row (#4768).

Tests: agc_driver_quad_pixel_mask, agc_driver_sample_mask_draw, agc_driver_sample_mask_depth_stencil, agc_driver_sample_mask_multisample, agc_driver_pixel_helper_lanes, agc_shader_quad_mask_no_demote; with this PR also request_serializer.

GPU Mode Base, ×3 This PR, ×5
RX 9070 XT (1002:7550), driver 32.0.31041.1004 GPL on quad_pixel_mask fails: implicit-LOD sample with quad pixel mask 2: pixel (0, 0) channel 2 is 2, expected 0; the rest pass all pass
RX 9070 XT GPL off all pass all pass
UHD 770 (8086:a780), driver 32.0.101.7088 GPL on all pass all pass

Unlike the 860M, this driver writes the demoted pixels only with GPL on. The values match the description: every uncovered pixel gets (0, 0, x, 0), with x = 2, 4, 8, 16 by column.

Full CTest on the RX 9070 XT with this PR, #6436 and #6647 together, from an empty VkCache: 1338 of 1339 (select_tests, symlink privilege on this machine).

AI-assisted: yes (Claude Code)

@adriadam10 adriadam10 self-assigned this Oct 18, 2026
@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Oct 18, 2026
@adriadam10
adriadam10 merged commit 1158cb8 into boykopovar:main Oct 18, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs shader core/shader unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants