fix(agc): accept RADV's disabled conservative rasterization and out-of-order watermark

[TheAggie]

What

RADV on the PS5 (mihawk-99/PS5_Mesa) programs two raster registers with values that mean "off" but that AnyPS5 rejected. ccleste's (PPSA59322) draws were rejected because of them.

Three commits:

1. PA_SC_MODE_CNTL_1 OUT_OF_ORDER_WATER_MARK. RADV sets the watermark to 7 on every GFX10 draw, also when OUT_OF_ORDER_PRIMITIVE_ENABLE is off. The watermark only tunes out-of-order rasterization, which stays rejected, so the field (bits 30:28) is accepted on its own.

2. PA_SC_CONSERVATIVE_RASTERIZATION_CNTL when disabled. RADV writes NULL_SQUAD_AA_MASK_ENABLE alone (0x100000), where libSceAgc writes the two INNER_TO_NORMAL overrides (0x6000). A value made only of those bits, including 0, now decodes as disabled. Overestimation (0x6001) keeps its checks, and every other mode is still rejected.

3. Point and line depth bias enable on triangle draws (added later, found running vkQuake, PPSA99010). With depth bias on, RADV sets POLY_OFFSET_PARA_ENABLE in PA_SU_SC_MODE_CNTL next to the front and back enables whatever the topology (vkQuake's draws carry 0x3a46). That bit applies the offset to points and lines, so it changes nothing for triangles, and Vulkan applies depth bias only to polygons. It is now accepted with depth bias on triangle topologies; point and line draws with it are still rejected.

Tests (agc_driver_graphics):

• A watermark of 7 without out-of-order rasterization is accepted, and with OUT_OF_ORDER_PRIMITIVE_ENABLE set the draw is still rejected.
• 0x6000, 0x100000 and 0 decode as disabled and are not rejected. Two enabled modes (0x00e00001, 0x01e00022) are still rejected.
• 0x3a46 on a triangle list keeps its depth bias and back-face culling; the same value on a point or line topology is still rejected. It fails with the change reverted.

Tested

Windows 11 Pro (26200), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release, NVIDIA GeForce RTX 5070 Ti.

• The full suite (488 tests) passes on current main with this PR and my FORCE_DST_ALPHA_1 PR merged together. It was run from the generated CTestTestfile.cmake commands because antivirus blocks ctest.exe on this machine.
• tools/check_conventions.py --base origin/main passes on this branch alone.
• For the third commit: agc_driver_graphics, agc_driver_pm4 and agc_driver pass on the pushed head; I didn't rerun the full suite for it.
• vkQuake (PPSA99010): its depth-biased draws were rejected without the third commit, on the same kind of local branch.
• ccleste (PPSA59322) renders with this change on a local branch that also carries my other open RADV PRs (feat(agc): accept graphics programs without an AGC shader header #1732, feat(agc): accept texture BC swizzles that leave the sampled border unchanged #1733-feat(agc): accept 8_8_8_8 UINT color targets #1736, feat(agc): accept the LINE_STIPPLE_TEX and POS_FIXED_PT pixel inputs #1819, feat(agc): run unregistered NGG vertex programs that need a workgroup on the mesh path #1825 and others).

Not tested on AMD, Intel, Linux or macOS.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

🤖 Generated with Claude Code

[klosixgaming-arch]

Small, correct, and I can approve it.

• PA_SC_MODE_CNTL_1.OUT_OF_ORDER_WATER_MARK: you accept bits 30:28 on their own (via ScanControlMask = ~0x76023fffu) while keeping OUT_OF_ORDER_PRIMITIVE_ENABLE rejected. The test asserts a watermark-7 draw is accepted and the same with out-of-order enable is rejected. Right boundary.
• PA_SC_CONSERVATIVE_RASTERIZATION_CNTL: decodeConservativeRasterization now accepts (control & ~0x00106000u) == 0 as disabled (covering 0x6000, 0x100000, and 0), keeps 0x6001 as overestimation, and still rejects everything else. The DrawRejection path matches the same mask, so decode and rejection agree.
• Both the decode (DecodeState) and the rejection (DrawRejection) paths were updated consistently — that's the easy place to desync and you didn't.
• Tests cover the three disabled values and two enabled values that must still be rejected.

One non-blocking note: the enabled-mode rejection test uses 0x00e00001 and 0x01e00022 — those are "enabled" conservative raster modes. Good. Just make sure the debt entry (if one is added later for "conservative rasterization enabled is unimplemented") records why 0x100000 is safe to treat as disabled (NULL_SQUAD_AA_MASK_ENABLE alone = off per RADV) so the asymmetry between "off" values and "on" values isn't a mystery to the next reader.

Written with AI assistance.

[johnmaia]

Linux run on AMD, since the description lists it as not tested: Ubuntu 24.04.5 (kernel 6.8), GCC 13.3, Release, AMD Radeon RX 7600 (RADV NAVI33, Mesa 25.2.8). This PR's head 88cc2ea4, merged onto main f1451396. Full ctest: 494/494 pass, the same as main on this machine, with the GPU tests running on the RX 7600 (none skipped). That includes agc_driver_graphics with this PR's new cases.

[MarcZX10]

The current upstream Windows run is red: agc_shader_disk_cache hit the 120 s CTest timeout; the Linux run and reported AMD Linux run are green. This does not point to the raster-state change, but CONTRIBUTING asks for the final checks to pass. Could you rerun the Windows job and confirm it clears before merge?

[Dextroid17]

Verified on Linux on lavapipe (Mesa) — the other side of the driver split this PR is about.

• Arm A (this PR): agc_driver_graphics_tests builds and passes, exit 0.
• Arm B (control): State.cpp back at main with the test kept → exit 1, and the message is precisely the register in question:

... sample iteration, primitive discard or out-of-order rasterization is unsupported (register 0x293 = 0x760201bc)

So the RADV value is exactly what the old code refused, and the new Requires are what catch it.

Claim cross-check: the scan-control mask widened from ~0x06023fff to ~0x76023fff (bits 30:28), so the out-of-order watermark is accepted on its own; the conservative control now accepts 0x6000 or RADV's 0x100000 via (control & ~0x00106000u) == 0. The part that matters is that the test checks the other direction too: out-of-order rasterization itself (0x7e0201bc) is still rejected, and genuinely enabled conservative values (0x00e00001, 0x01e00022) are still refused through expectFailure. So this accepts a value that means "off" without accepting the feature.

Citable source is in the body (mihawk-99/PS5_Mesa, the RADV-on-PS5 port), which is what a quirk acceptance needs.

Not verified: the RADV path itself — there is no AMD GPU or PS5 Mesa here, so lavapipe only shows that the change does not regress the Mesa path — and Windows.

Written with AI assistance.

[EdouardSence]

Another disabled value from a retail title, which this PR would still reject: Ratchet & Clank: Rift Apart (PPSA01474) writes PA_SC_CONSERVATIVE_RASTERIZATION_CNTL = 0xe6000 on its first draws. On main plus the fixes it needs to get there (#1673, #1339, #1836, #1893, #2105, #2184), the run stops about 19 s in on:

AGC graphics: PA_SC_CONSERVATIVE_RASTERIZATION_CNTL=0xe6000: only 0x6000 (off) and 0x6001 (overestimation) are supported

Decoded with the field shifts in the kernel's gc_10_3_0_sh_mask.h:

Bits	Field	Value
0	OVER_RAST_ENABLE	0
5	UNDER_RAST_ENABLE	0
13, 14	OVERRIDE_{OVER,UNDER}_RAST_INNER_TO_NORMAL	1, 1 (as in libSceAgc's 0x6000)
17:16	UNCERTAINTY_REGION_MODE	2
18	OUTER_UNCERTAINTY_EDGERULE_OVERRIDE	1
19	INNER_UNCERTAINTY_EDGERULE_OVERRIDE	1

So both rasterization enables are off, like 0x6000. The extra bits only set the uncertainty-region mode and its edge rules. RADV never writes those fields; it sets PBB_UNCERTAINTY_REGION_ENABLE only for overestimation. I have no source that says the uncertainty fields do nothing when both enables are off, so I'm not sending a fix based on that guess.

If you think it fits this PR, would you consider treating any value with OVER_RAST_ENABLE and UNDER_RAST_ENABLE clear as disabled, with the uncertainty-region fields recorded in TechnicalDebt? Or tell me whether you'd rather keep it to RADV's value. I can test a candidate on this title.

Measured on Linux, RX 9070 XT. Tested and written by Claude (AI assistant) on a contributor's machine.

[Dextroid17]

Your restraint is the right instinct and I would not overrule it. With no source for what the uncertainty-region fields do when both enables are clear, accepting 0xe6000 as "off" is a guess — and a guess that renders wrongly is worse than a title stopping with a message that names the value. That is the standard this project applies elsewhere, and you are applying it to your own convenience.

If it does get accepted, the shape you propose is the narrowest version: treat a value as disabled only when OVER_RAST_ENABLE and UNDER_RAST_ENABLE are both clear, and record the uncertainty-region mode and its edge rules in TechnicalDebt rather than asserting they are inert. Whether that belongs in this PR is the maintainer's call, not mine.

What would make it easy for them to say yes is the test you offer. If the candidate renders correctly on Rift Apart in the cases where overestimation actually matters, that is evidence rather than an argument — and if you run it, please say what the candidate was and what changed in the frame, not only that it worked.

Written with AI assistance.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#1924	core/libs/prx/libSceAgcDriver/tests/Graphics.cpp L913
#2322	core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp L572-573