AGC: pixel shaders that export a stencil reference value abort on devices without VK_EXT_shader_stencil_export (NVIDIA; Demon's Souls, 1080p renderconfig)

Demon's Souls (PPSA01341 v1.07, US) aborts during the tutorial level load with:

[videoout] presentation failed: depth export, shader coverage or ordered fragment execution is unsupported (register 0x203 = 0x00000052)
terminate called after throwing an instance of 'std::runtime_error'

Register 0x203 is DB_SHADER_CONTROL. 0x52 sets three bits:

bit 1, STENCIL_TEST_VAL_EXPORT_ENABLE: the pixel shader exports a stencil reference value
bits 4-5, Z_ORDER = 1 (early Z then late Z)
bit 6, KILL_ENABLE

ShaderControlMask in Graphics/src/State.cpp already accepts bits 4 to 6. Bit 1 is the only one it rejects, so this is a draw whose pixel shader writes the stencil reference. There is no stencil export path today: nothing in the driver or the recompiler emits FragStencilRefEXT or enables VK_EXT_shader_stencil_export.

How to reach it: the title only takes this path here when it renders at 1920x1080. Following ponhei9502's note in #1283, I set +r_Wants4K = WantsPerf in copies of the five app0/misc/renderconfig_ps5_*.txt files (a local copy of app0/misc, the dump untouched) and booted through the debug start menu into the tutorial. The abort came at about 1209 s, during level loading, after 157 presented frames. With the default renderconfig (4K) the same build gets to a different stop first (a depth plane sampled as a texture), so I haven't seen this draw at 4K yet.

I don't have the pixel shader address. The check fails while the draw's state is decoded, and the message doesn't name the program. I can add that and report back if it helps.

Setup: Linux, RTX 5070 Ti, NVIDIA 595.91.07. main 7a84b22 with #4049, #758, #4241 and #4265 merged locally (#4265 is the fiber stack fix that gets this far), ANYPS5_ENABLE_SPIRV_TOOLS=ON, APS5_SPIRV_OPT=none.

AI-assisted: yes

fix(agc): support stencil reference export from pixel shaders

Status on main: the stop in this issue's title is gone on devices with VK_EXT_shader_stencil_export. #4355 (merged 2026-10-12, 22d662eda) emits FragStencilRefEXT for pixel shaders with DB_SHADER_CONTROL.STENCIL_TEST_VAL_EXPORT_ENABLE. It says "Refs #4276", so this issue was not closed with it.

On AMD the new test passes: agc_driver_stencil_ref_export passes 5 of 5 (not skipped) at main a0f5916a2. Setup: Windows 11 (26300), Radeon 860M (RDNA 3.5, driver 32.0.13058.8001, which advertises VK_EXT_shader_stencil_export), WinLibs GCC 15.2.0, Release.

What is still open is NVIDIA, whose driver lacks the extension. There the draw stops with a pixel shader that exports the stencil reference needs VK_EXT_shader_stencil_export, which the device lacks. #5304 removes that stop for draws without a stencil test, but Sam-mon7's Demon's Souls draw on the same PR has the stencil test on. It may be clearer to retitle this issue to that NVIDIA case, or to close it in favour of a new one.

AI-assisted: yes (Claude Code)

@LeandroLP23

@LeandroLP23 thanks, retitled to the NVIDIA case. #4355 covers devices with VK_EXT_shader_stencil_export; the RTX 5070 Ti here doesn't advertise it, so Demon's Souls still stops on this draw. #5304 and #5466 both target that remaining case; I'll test them on Linux NVIDIA with Demon's Souls and post the results here.

AI-assisted: yes

@Sam-mon7

Results for #5304 and #5466 on Demon's Souls, as promised above. Neither gets the title past this stop on NVIDIA yet.

Setup: Linux, RTX 5070 Ti, NVIDIA 595.91 (no VK_EXT_shader_stencil_export). Demon's Souls PPSA01341 v1.07, 1080p renderconfig, debug direct boot. Base is main 7a0c838 plus a local merge of #758, which the title needs to reach the level load. ANYPS5_ENABLE_SPIRV_TOOLS=OFF.

build	result
`main` only	stops earlier, at t+1185 during the level load: `GetImageResource dword 0` (#758's case)
`main` + #758	t+1471: `a pixel shader that exports the stencil reference needs VK_EXT_shader_stencil_export`, shader 0x24ab8e200
+ #5304	not run. Every stencil-export draw we logged has the stencil test on (`DB_DEPTH_CONTROL` 0x007007e7 / 0x007007c7), so its rule doesn't apply
+ #5304 with @Sam-mon7's widening (no face consumes the reference)	the first draw passes. t+1301: same error on shader 0x24ab6ae00, a draw that writes the reference (`DB_STENCIL_CONTROL` 0x00030030, ZPASS REPLACE on both faces, `DB_STENCILREFMASK` 0x018080ff, write mask 0x80)
+ #5466 (`c7595e9`)	t+474: #5466's refusal `stencil reference export emulation requires a replay-safe pixel shader with late tests`, shader 0x24abb6d00
+ #5466 + the widening	the widening takes 0x24abb6d00 out of replay, then the same refusal on 0x24ab57200 (t+561)
+ #5466 with fdfawef324's three #758 adaptations	same refusal, shader 0x24ab6dc00

The refused addresses run the same 788-word pixel shader (0x24abb6d00 and 0x24ab57200 are byte-identical). The draws differ only in stencil state: one doesn't consume the reference, so the widening drops its export, and the other writes it. Why #5466 refuses that shader is on #5466.

AI-assisted: yes

@fdfawef324

Another NVIDIA case, from Onimusha: Way of the Sword (PPSA27836) during its first scene load. RTX 5070 (617.42), Windows 11, main 17cfb1f + #6762 (STENCIL_OP_VAL_EXPORT with REPLACE_OP, value in MRTZ.Y bits 8-15):

shader 0x2107d4b600 stage 5: ShaderRecompiler::Recompile: stencil reference export emulation requires a proven set of at most 16 values

The draw: DB_SHADER_CONTROL 0x45 (Z export, operation-value export, kill), DB_DEPTH_CONTROL 0x7007e7, DB_STENCIL_CONTROL 0x40040 (REPLACE on depth pass, both faces), DB_STENCILREFMASK write mask 0xff.

#5466's replay can't take this shader even with a larger value limit, for two reasons:

The value is either a user-data byte (s4 & 0xff) or comes from a buffer read.
The shader has side effects: a global atomic and buffer stores before exp mrtz.

So replaying per value would run those side effects more than once.

An idea for draws like this one, where the only stencil op that uses the reference is REPLACE on depth pass and the stencil test is ALWAYS. I haven't tried it:

Run the draw once. Write the 8-bit value plus a "written" flag to an extra R8G8 color attachment instead of FragStencilRef. With ALWAYS, a fragment that passes the depth test is exactly one whose stencil would be replaced, so the last fragment written per pixel is the same one the console keeps.
Then set the stencil from that image with a full-screen pass per bit: discard where the flag is clear or the bit is 0, REPLACE with reference 1 << b and write mask 1 << b, after one pass that clears the masked bits where the flag is set.
That is 9 cheap passes, and the shader runs only once.

Before anyone writes it: does this match how others see the problem? I'd like to hear from @fdfawef324 (#5466) whether this kind of second path belongs next to the replay or should replace part of it.

AI-assisted: yes (Claude Code); the run and the decoding are mine.

fix(recompiler): replay every eight-bit stencil reference a pixel sha…

While working on the recorder test registrations I ran every flag-scoped mode of the AGC
driver against two devices, and one of them found a case of this. It is not a new symptom,
but it locates the failure and shows why it is invisible today.

Reproduction

The test is agc_driver_stencil_ref_export_unused_tests, in
core/libs/prx/libSceAgcDriver/tests/execution/StencilRefExportUnused.cpp. It takes a
--case argument, and six of its eight cases fail on every device:

`--case`	NVIDIA RTX 4060, driver 615.78.08	Intel Iris Xe, Mesa 26.2.4 ANV
`keep`	0	0
`invert`	0	0
`compare`, `back-compare`, `replace`, `fail-replace`, `depth-fail-replace`, `partial-replace`	1	1

The failure message is the test's own:

AGC graphics: a dependent reference without native stencil export was admitted

The two devices differ in the extension, and the test skips on one of them

VK_EXT_shader_stencil_export is present on the Intel device and absent on the NVIDIA one.
The test's keep and invert cases pass on both, and the six dependent cases fail on both.

The reason the Intel run skips rather than fails is in the test's own expectation: it looks
for the refusal

a pixel shader that exports the stencil reference needs VK_EXT_shader_stencil_export, which the device lacks

and on the Intel device the draw is admitted for a different reason, so the expectation does
not hold there either.

Where the refusal and the emulation live

The message the test expects is thrown in
core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvModuleSetup.cpp:380, and it is
reached only on the native path, that is, when state.stencilRefVariable != 0 and
state.stencilReplay is false. state.stencilReplay is set in
core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvEmitterHelpers.cpp:425, in the
branch that runs when the device does not have the capability, and that branch emulates the
stencil reference export instead of refusing it.

So on a device without VK_EXT_shader_stencil_export the driver emulates the export, and
the refusal the test expects no longer happens. The emulation has three preconditions of its
own, each with its own message, and none of them fires for these six cases: the draw is
admitted silently.

Whether the emulation is right for a dependent stencil reference, or whether the draw has to
be refused as the test expects, is a question about the console's own behaviour, which I
cannot measure. I did not touch the driver.

Why this is invisible

agc_driver_stencil_ref_export_unused_tests is built, because the AGC execution tests are
globbed from tests/execution/*.cpp, but no ctest entry runs it, and no ctest entry passes
it a --case either. The six cases are therefore never executed by CI, and a local run only
reaches them with the argument by hand.

The registrations I am proposing in #8445 add the two cases that pass, keep and invert,
and leave the six that fail out, so that the coverage grows without a red job.

In-title gate map for Assassin's Creed Shadows (PPSA20397) — stencil-export replay, end to end.

RTX 3070 (driver 617.42, no VK_EXT_shader_stencil_export), Windows 11, VK_LAYER_KHRONOS_validation 1.4.363 active, ANYPS5_NO_SHADER_CACHE=1. Stack: #9317 92ebac24 + #9393 + #9904 + #9905 + #9906 + #8615 15803c7f own commits, full 367-module .prx set rebuilt and deployed to the title's libs/.

The game now boots and renders ~5 minutes before reaching the stencil draw. Working through the gates in order:

Recompile: ... requires a replay-safe pixel shader with late tests — cleared by the fix(recompiler): exempt proven control conditions and own-bit ballots in stencil replay #9317 chain. 0x407325e400 prepares at use (+278s) and enters replay.
stencil replay requires single-sample rendering and an eight-bit stencil format — the stencil-exporting draw is multisampled; the stencil-format half passes. This run split the Require: it's the state.samples == 1u half that fires. A local relaxation (dropping only the samples check) let the draw proceed — worth evaluating whether per-sample stencil test+replace passes are actually invalid under MSAA, since the exported reference is per-fragment.
guest buffer offset lies outside the shader's push constants — on a different stencil shader (0x407321e400, unbounded 256-reference export) in an earlier run; this is exactly the bug fix(agc): put a stencil replay shader's guest buffer offsets where its shader data is #8615 fixes. With fix(agc): put a stencil replay shader's guest buffer offsets where its shader data is #8615 in the stack that shader did not recur this run (draw ordering isn't deterministic), so in-title confirmation of its fix path is still open — but no regression was observed.
stencil replay refuses sampled depth-surface feedback — the current frontier. 0x407325e400 samples the depth surface its draw renders into (log line a draw replays all 256 stencil reference values fires, then this refusal). Every replayed pass re-runs depth, so a sampled depth target is genuine feedback for the replay mechanism even if the console tolerates it. Likely needs a depth-plane snapshot bound for sampling during replay, or a narrower check that only refuses when the replayed passes can change what the shader samples.

Each gate needed the previous ones cleared to be visible — this list is what stands between PPSA20397 and a drawn stencil-export pass on a non-stencil-export device.

Gate 2 analysis — why state.samples == 1u looks unnecessary (stencil replay under MSAA).

Following the in-title run, I split the Require and confirmed ACS's stencil-export draw is multisampled while its stencil format is fine. After removing only the samples check locally, the title advanced past replay setup into the next (orthogonal) gate — so the mechanism itself works under MSAA. The semantics also hold on inspection:

The replayed variant stores the shader's computed stencil value into stencilRefVariable at exp mrtz; at function end it compares it against the pushed reference and OpKills the fragment on mismatch (SpirvFlowEmitter state.stencilReplay tail). Each of the up-to-256 passes is then an ordinary draw of the title's own stencil ops with VK_DYNAMIC_STATE_STENCIL_REFERENCE = r.
discard under multisampling zeroes the fragment's covered samples; stencil ops apply per sample. So pass r writes r to exactly the samples of fragments whose computed value is r — the same set the console produces.
There is no per-sample stencil export to lose: the GCN MRTZ stencil channel carries one value per pixel; hardware cannot express sample-varying stencil references, so nothing stencilReplaySafe could admit would diverge per sample.
The recording path already composes the two: record.stencilReplay and the multisample/sample-order bookkeeping (record.multisampleTargets, record.reordered, reorderPass) coexist in RecordedDraw — recordStencilReplay runs between the order pass and the resolve store, so MSAA resolves and the sample-order pass apply to the recorded passes as usual.

So the state.samples == 1u half of Draw.cpp's stencil-replay Require guards a case the design already handles. A sound relaxation is Require(!state.depth || HasStencil(state.depth->format)) alone — possibly worth keeping an eye on pipeline->ReordersSamples() + replay interaction on the GPUs that path exists for, since reorder runs outside the per-reference passes.

Happy to shape this into a PR (with an agc_driver_stencil_replay MSAA case) once the open chain it's measured on settles — noting it here so it isn't rediscovered a third time.

Gate 4 root-caused and cleared locally — the sampled-depth refusal is over-broad.

Instrumented the DepthSurfaceAt check for 0x407325e400's draw:

sampled T#s:  base=0x41b3290000 1864x1048 fmt=22
              base=0x41b3b00000 1864x1048 fmt=5
bound target: depth=0x41d4800000 stencil=0x41d5070000 932x524 fmt=130

The shader samples two previous-pass depth surfaces (TAA/reprojection-style readback) — not the surface it renders into. DepthSurfaceAt resolves the T# against any active surface in the registry, so a read of an unrelated, stable depth buffer trips the feedback guard. The neighboring ReadsOverlap(state.depth->address, ...) checks are already bound-target-scoped; the texture path is the outlier.

Local fix that clears it: a BoundDepthSurfaceAt(context, words, texture, bound) variant that only matches when the found surface's target.address equals the bound state.depth->address (same record → same surface → genuine feedback; different surface → provably stable across replay passes, since replay only touches the bound attachments). With it, ACS advanced to gate 5.

Gate 5 (new, and already-covered territory):

AGC graphics: a CPU read of 0x41b1940000+0x200000 overlaps multisampled color target
0x41b1800000+0x870000 (4 samples), whose samples are only on the host: guest access to
it is not implemented

That's the general "guest access overlaps a native MSAA color target" problem — #9186 (retire an undrawn filled target a guest access overlaps), #8223, #9892 all work this area, so this probably isn't new work. From here the stencil chain joins the ordinary multisample-target queue rather than needing stencil-specific fixes.

Chain state for PPSA20397: gates 1 (replay-safety) → union stack, 2 (MSAA) → sound to relax per my previous comment, 3 (push constants) → #8615, 4 (sampled depth) → bound-scope the check, 5 → likely #9186-class.

Gate E update — validated open #9186 against the ACS failure: it does not clear it.

At +486 s (0x407325e400, 256-reference replay, bound-depth check passed) the title still stops at a CPU read of 0x41b1940000+0x200000 overlaps multisampled color target 0x41b1800000+0x870000 (4 samples), whose samples are only on the host. The target's hostOnly flag is set — it was rendered to, not filled-for-sampling, so #9186 correctly refuses retirement. Its design works as intended; ACS is the drawn-target half of the space. The remaining gap is guest access to host-resident samples — #9892's direction (host-resident PS4 targets / resolve-back), or a readback path for CPU reads over drawn MSAA targets. Full datapoint on #9186.

Windows NVIDIA data point for PPSA01342 (US) v1.005: RTX 3080 10 GB, driver 616.56 (no VK_EXT_shader_stencil_export), Windows 11 Pro 26200, main 5d3ab7791, renderconfig unchanged. Loading a save in the second world (Stonefang Tunnel) stops at the same draw in two runs out of two, one on plain main and one with #10632/#10633 (CPU-side only):

[Shaders/ShaderRegistry.cpp:901 ReportPreparedAtUse] Shader 0x24a7a3200 stage 5 has no artifact prepared at registration for the state of this draw or dispatch; preparing it at use
[gpu] driver failure: shader 0x24a7a3200 stage 5: ShaderRecompiler::Recompile: stencil reference export emulation requires a replay-safe pixel shader with late tests

The full RecompileRequest of that shader is in #10634 (comment). Other runs from the same save hit a device loss (#10634) before reaching it.

#10351 targets this refusal for late-test draws. I can build it (with #6782) and run it on this save on the RTX 3080 if that would help; I'll post the result either way.

🤖 Generated with Claude Code

ACS PPSA20397 update — the D32S8 stack clears the +848s blocker; the next wall is a new MaskReplaySafety case.

RTX 3070 (617.42, no VK_EXT_shader_stencil_export), Windows 11. Stack: our val chain + merged #6300/#9533/#9717 (all 19 commits of pr9717) + the full mask chain (#9317, #9393, #9030, #9266, #9904, #9905, #9906 — verified patch-identical to current PR heads).

Gate progress since my Oct-23 gate map:

0x40054ea100 compute (native multisample stencil/depth import) — gone. The D32S8 separate-stencil-plane import works; the title advances past the former +848s crash point into pixel shading it never reached before.
New stop, ~+10–12 min: 0x407335e400 (earlier run 0x407325e400, same 624-word shader body at a different address), stage 5:

ShaderRecompiler::Recompile: stencil reference export emulation requires a replay-safe pixel shader with late tests

Request decode: stencilRefExport=1, stencilOperationValueExport=0, psEarlyZ=0, psOrderedPixelShader=0, iteratesSamples=1, wave64, StencilReferenceValues = 256, 4 captured memory regions, 2 linked programs.

Instrumented diagnosis: StencilReplaySafe itself fails on MaskReplaySafety.Check() — specifically maskStable() on the shader's own-lane mask read (own mask unstable: the (ballot >> (lane & 31)) & 1 word can't be proven stable across the replay passes). No loop headers, no BDA writes, no side-effect opcodes — the mask-stability proof is the only blocker. The full open-PR mask chain is present, so this is a case none of #9904/#9905/#9906 currently covers.

Worth noting: the same shader address entered replay on my Oct-23 stack (where the D32S8 surface failed to import). The plausibly-changed input is the draw's stencil state — with the stencil plane now properly bound, the IR/mask structure the shader presents differs. If so, replay acceptance is sensitive to the draw's D/S state, which may matter for reproducing.

The failing .req (shader_407335e400.req, replays the throw under --prepare) is in the ACS dump; happy to share it or run further instrumented probes.

AI-assisted: yes

Navigation Menu

Repository navigation

AGC: pixel shaders that export a stencil reference value abort on devices without VK_EXT_shader_stencil_export (NVIDIA; Demon's Souls, 1080p renderconfig) #4276

Reproduction

The two devices differ in the extension, and the test skips on one of them

Where the refusal and the emulation live

Why this is invisible

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

AGC: pixel shaders that export a stencil reference value abort on devices without VK_EXT_shader_stencil_export (NVIDIA; Demon's Souls, 1080p renderconfig) #4276

Description

Activity

LeandroLP23 commented on Oct 13, 2026

fisherjoey commented on Oct 13, 2026

fisherjoey commented on Oct 14, 2026

simonbronk commented on Oct 17, 2026

juanpimg commented on Oct 20, 2026

Reproduction

The two devices differ in the extension, and the test skips on one of them

Where the refusal and the emulation live

Why this is invisible

akkadbakkad89 commented on Oct 23, 2026

akkadbakkad89 commented on Oct 23, 2026

akkadbakkad89 commented on Oct 23, 2026

akkadbakkad89 commented on Oct 23, 2026

noir39 commented on Oct 24, 2026

akkadbakkad89 commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions