perf(agc): allow shader preparation on first use

[Sam-mon7]

What

Adds opt-in APS5_LAZY_SHADER_PREPARE=1: shaders are still validated and decoded at registration, but prepared on their first draw or dispatch instead of all at boot. The default is unchanged.

Why: Demon's Souls (PPSA01341) registers thousands of shaders during boot. Preparing all of them up front pushes private memory to 17 GB and runs a 32 GB Windows machine out of RAM before the first real frame. #5806 reports the same on main under an 18 GiB cap.

Demon's Souls boot, same machine and scene	Peak private	Lowest available RAM	Result
main + #5870	17.1 GB	172 MiB	Stopped at frame 2 (out of RAM)
main + #5870 + this PR, opt-in on	12.2 GB	7.8 GB	Reaches the intro cutscene

(#5870 fixes an earlier abort on main; without it the game stops before this point. This PR doesn't depend on it.)

Tested

Windows 11 (26200), WinLibs GCC 15.2.0, Release, RTX 4070.

1. Demon's Souls with and without the opt-in: the table above. The out-of-RAM stop is a guard that ends the run below 768 MiB of available RAM for 30 s, before Windows starts paging the system to a halt.
2. agc_driver_prepared_shaders*: 12 passed, including the new lazy and lazy_sync modes. These cover dispatch, metadata and failure reporting on first use.
3. ^(agc_|windows_backtrace_): 534 passed, 0 failed, 3 skipped (two udmabuf tests are Linux only; the RTX 4070 has no stencil export).

PR overlap: #5115 edits a nearby line of tests/execution/PreparedShaders.cpp. Either can merge first; the rebase is trivial.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA01341 Demon's Souls (PPSA01341) #1283
• GPUs tested: NVIDIA GeForce RTX 4070, driver 596.21, Windows 11
• AI-assisted: yes

[CnRJay]

The memory numbers are a real problem for Demon's Souls. Before an opt-in mode, though, it is worth knowing what the extra 5 GB is.

In lazy mode, the first draw or dispatch of each shader goes through SourceHandleFor → PrepareShaderWithDiagnostics on the thread recording it, while it holds snapshot.prepared->mutex. With a cold disk cache that is a full recompile (SPIR-V generation and optimization) on the queue worker. With a warm one it is a disk-cache load and parse. Both stall the queue in the frame where the shader first appears, which is the stutter the background registration pool was added to avoid (#2583). The PR measures memory only, so the cost on the other side isn't visible. The summary lines of one scene with the opt-in off and on, warm cache, with frame_ms p99/max, would show it, for Demon's Souls or a title that doesn't run out of memory.

On the memory itself:

• If most of the 17 GB is prepared SPIR-V or artifacts held for shaders that are never drawn, a smaller default for the prepared-artifact cap or evicting prepared entries by last use would fix it for every user without a switch.
• If it is the queued RegisteredPreparations, #5910 removes the decoded program each one holds, which was 638 MiB in Borderlands 4.

A heap profile at the 12 GB / 17 GB points would show which of these it is. As it stands, Demon's Souls still runs out of memory by default, and users would need to know about the variable.

[AnthonyClemens]

Tried this on Resident Evil 2 (PPSA04288), Windows 11, RX 9070 XT, 32 GB. Local integration tree on main 93d69ed; the only difference between the runs is APS5_LAZY_SHADER_PREPARE. Same build, warm shader cache (~8,000 shaders), 300 s boot through the first-boot setup screens, private bytes sampled every 10 s:

	peak private	private on the setup screens	fps there
opt-in off	15.2 GB	14.8-15.2 GB	6.6
opt-in on	14.3 GB	13.9-14.2 GB	6.7

So about 1 GB less here, with the same frame rate and the setup screens reached a bit sooner (161 s vs 170 s). Smaller than on Demon's Souls, because most of RE2's footprint at that point is its own direct memory rather than prepared shaders, but it's a clear win with no downside I could see. I didn't reach gameplay in these runs.

[Sam-mon7]

Thanks, this was the right question. I measured it, and the answer points at your first option.

Short version: with the opt-in off, registration loads the prepared artifact of every registered shader at boot, about 6,300 shaders, while the game has drawn only ~170 by the intro. The extra memory is those undrawn artifacts, and it doesn't show up as private heap.

Demon's Souls (PPSA01341) boot, Windows 11, RTX 4070, 32 GB RAM, main 2559e9623 + #5870 + #5910:

Run	Peak private	Peak working set	Disk-cache hits at boot	Result
Opt-in off	14.4 GB	16.4 GB	5,654 (6.0 GiB read)	Out of RAM before the first scene
Opt-in off, APS5_SYNC_SHADER_PREPARE=1	13.1 GB	17.6 GB	6,311 (7.0 GiB read)	Out of RAM before the first scene
Opt-in on	12.9 GB	10.7 GB	171 (0.3 GiB read)	Reaches the intro with ≥3.4 GB RAM free

What this rules in and out:

1. Not the queued decoded programs: perf(agc): free a registered shader's decoded program once it is planned #5910 is applied in every row and the opt-in-off runs still run out of memory.
2. Not pool concurrency: synchronous preparation runs out of memory too, so the cost is what is kept after preparing, not compiling in parallel.
3. Not mostly private heap: private memory is about the same in all three runs. The difference is working set, which is consistent with the cache entries mapped since fix(shader): map retained SPIR-V from shader cache entries #4806 for every registered shader. A heap profile wouldn't show it.

So a cap, or evicting prepared entries by last use, should fix this for everyone with no switch, as you suggested. I'll look at that as a follow-up and close this PR once it lands, unless you'd rather it closed now.

On the stutter cost: I can't give frame_ms with the opt-in off for Demon's Souls, because it doesn't get past boot on this machine. You're right that lazy preparation moves the work onto the queue worker. I'd rather fix the retention than trade it for first-use hitches.

[adriadam10]

Checked the diff, built it merged onto main and ran the AgcDriver ctest scope (only agc_driver_tessellation fails, it segfaults on main too).

• tests/execution/PreparedShaders.cpp:623 (LazyRegistration, mode lazy): it passes with the ShaderRegistry.cpp and docs change reverted, so it doesn't test the opt-in. It only checks dispatch output and error reporting, which eager preparation also gives. Please assert something only lazy mode does, e.g. that registration leaves the snapshot deferred (no prepared artifact / no disk-cache load until the first dispatch), and show it fail without the change. lazy_sync does fail without it.
• @CnRJay's first-use stall question is still open: frame_ms p99/max with the opt-in off and on, warm cache, on a scene that doesn't run out of memory.
• Your own measurements point at retention (cache entries mapped for every registered shader), and you said a cap or eviction by last use would fix it without a switch. If that is the plan, this env var is a stopgap for one title; say whether you want this merged meanwhile or closed in favour of the follow-up.

The memory numbers (Demon's Souls 17.1 → 12.2 GB, RE2 15.2 → 14.3 GB by @AnthonyClemens) are clear. Happy to re-review once the test is fixed.

[Sam-mon7]

Thanks for the review. All three points are addressed below.

1. The lazy test now checks the opt-in (f515ed29f)

• After registration it reads the published snapshot and requires it to be deferred with no prepared artifact. After the first dispatch it requires a prepared artifact.
• To read that state, Driver::RegisteredShaders() returns the registry under the driver's mutex, like the existing registry copies in ShaderRegistry.cpp.
• With the ShaderRegistry.cpp change reverted, both agc_driver_prepared_shaders_lazy and lazy_sync fail with "lazy registration prepared the shader before its first dispatch". With it, all 11 agc_driver_prepared_shaders* cases pass.

2. First-use stall: Demon's Souls' intro, warm cache, same build (main 69e7138ac + #5870 + this PR), APS5_HOST_IMPORT_MIB=8192 and APS5_TRACE_SUMMARY_EVERY=10 on both sides, no other switches, RTX 4070, Windows 11.

21 intervals of 10 s after boot	Opt-in off	Opt-in on
frame_ms p50 (median of intervals)	~295	~275
Intervals with frame_ms max > 1 s	5 (worst 2873)	0 (worst 875, in the first interval)
Peak private / working set	27.8 GB / 10.2 GB	12.8 GB / 10.0 GB

On this scene the opt-in added no visible hitches. The caveat: with the opt-in off this machine pages (27.8 GB private in a 10.2 GB working set), so some of the off-side spikes are paging. I don't have a scene where the opt-in off fits in RAM here, so this doesn't fully isolate first-use preparation cost on a machine with headroom.

3. Stopgap or close: I'd like this merged as an opt-in stopgap. It's off by default, and for Demon's Souls and RE2 it's the difference between booting and running out of memory. Meanwhile I'll work on the general fix you described (bounding or evicting prepared entries by last use) as a separate PR, and close this one if that change covers the same case without a switch.

[fdfawef324]

Demon's Souls (PPSA01342), measured into the level rather than boot only. Windows Server 2025 (26100), RTX 3060 (driver 591.86), Ryzen 9 5900X, 32 GB RAM, 64 GiB pagefile (commit limit 95.9 GiB). Each run is 40 min: boot, the debug walk-through menu, Tutorial (Fort Entrance) picked by scripted pad input, the level load, then idle in Outpost Passage. Memory is sampled every 10 s, in GiB, one run per row.

The build is not main: it is upstream 0214069e3 plus a private stack of about 90 commits. The stack carries this PR's runtime change (cherry-picked; its ShaderRegistry.cpp hunk is identical to 9607018, and c8bedff only adds the test accessor), #5910, and the pre-merge opt-in revision of #4806 (APS5_FILE_BACKED_SPIRV=1: SPIR-V modules of 64 KiB or more go to temporary file-backed views, and the rest stays on the heap). It also has a byte cap on those views, APS5_FILE_BACKED_SPIRV_MAX_BYTES, from a local version of the bounded-storage correction we offered on #4806, which was not merged and is not in any PR; above the cap, SPIR-V stays on the heap. A and C are one binary; B, D and E are that binary plus #5484. B to E set APS5_JOB_AFFINITY_MASK / APS5_WORKER_AFFINITY_MASK; A does not.

	Opt-in; other env	Shaders prepared¹	Peak private	Private at menu² / in level³	Peak commit⁴	Pagefile in use, start → max	SPIR-V in file-backed views at exit
A	off; cap 8 GiB, no affinity masks	9,781	46.9	19.7 / 44.4	76.9 / 59.8	5.0 → 27.4	8.0 (at cap)
B	off; cap 32 GiB	9,781 (4,642 cold)	52.4⁵	20.1 / 40.8	76.7 / 61.4⁵	6.7 → 28.5	12.3
C	on; as B	1,374 (1,052 cold)	25.8	8.5 / 23.0	53.3 / 38.0	6.8 → 7.2	1.7
D	on; as B + APS5_NO_LIBRARY_PREBUILD=1	1,374	26.3	8.3 / 23.2	53.3 / 38.3	7.0 → 8.2	1.7
E	on; as D + APS5_HEAP_CACHE_MIB=512	1,415	26.0	10.6 / 23.7	53.3 / 38.1	6.8 → 7.5	1.7

¹ Shader disk-cache hits plus misses over the run; "cold" is the misses. ² At the final Cross on the walk-through menu. ³ Mean from the first frame dump showing the level HUD to the end of the run. ⁴ System commit / system commit minus other processes' private bytes (other processes held 15–17 GiB at each run's commit peak). ⁵ Set by short spikes at 1,320–1,660 s, about 15 min after B's last cache miss was compiled (all 4,642 were done by about 450 s). We haven't identified the cause, and A has no such spikes. Five single 10 s samples sit 4–9 GiB above their neighbours, and one of them sets the commit peak. Peak private is the process's peak-commit counter, which also caught larger transients between samples; no 10 s sample of B's private bytes is above 48.7 GiB. Without the five spike samples, B's highest private sample is 43.5 GiB and its peak commit is 71.3 / 56.3 GiB.

• The opt-in (B→D: same binary, cap and affinity masks; D also sets APS5_NO_LIBRARY_PREBUILD=1, which adds nothing, see below): private memory in the level is about 18 GiB lower (40.8 → 23.2), and peak commit minus other processes is 18 GiB lower (56.3 → 38.3, without B's spikes). B→C changes only the opt-in in the environment, and the binaries differ only by fix(agc-driver): keep a live-backed snapshot mismatch from failing the dispatch #5484. It gives the same result: 17.9 GiB in the level and 18.4 GiB of commit. A→C (one binary) shows 21–22 GiB, but C also raises the cap and adds the affinity masks. A→B puts the cap's effect at about 3.5 GiB in the level, which matches the 4.3 GiB of extra SPIR-V that went into file-backed views.
• Paging: with the opt-in off, the pagefile grew by about 22 GiB and available RAM fell to 0.11 GiB (A) and 0.30 GiB (B). With it on, the pagefile grew by 0.5–1.1 GiB and available RAM never went below 1.18 GiB.
• What eager preparation keeps: A and B both prepared 9,781 shaders. With the opt-in, 1,374–1,415 had been prepared by the end of the run, after more than 20 min in the level. SPIR-V in file-backed views (not counted in private or commit) fell from 12.3 to 1.7 GiB, and the ~18 GiB private difference is on top of that. So on this stack, most of what eager preparation keeps in the level is private memory, not the large SPIR-V in file-backed views. That may matter for the cap/eviction follow-up. That private figure includes SPIR-V modules under 64 KiB, which stay on the heap here. We didn't measure them separately or run eager with APS5_NO_LIBRARY_PREBUILD=1, so we can't say how much of it is small SPIR-V or the registration-time library prebuild.
• APS5_NO_LIBRARY_PREBUILD=1 (perf(agc): allow disabling graphics library registration warmup #5551) adds nothing on top of the opt-in (C→D: +0.5 GiB peak, +0.2 GiB in the level, same peak commit; C, D and E differ by up to 0.7 GiB in the level). With the opt-in, registration discards the plan (ShaderRegistry.cpp:1522-1525 at c8bedff), so the registration-time prebuild in PrepareRegistered (:900-902) never runs. The knob then only gates prebuilds reached from explicit ABI resolution (ResolveShaderAbi at :1308, ResolveGraphicsStagesAbi at :1262). APS5_HEAP_CACHE_MIB=512 (E) made no measurable difference either.
• Time: the game's own walk-through timer ("Automation completed, took") read 157–182 s with the opt-in, against 299 s (A) and 417 s (B). The in-level present rate was the same in all five runs (0.32–0.35/s). Neither pair isolates load time, because A has no affinity masks and half of B's cache was cold. At about 0.33 presents/s, these runs can't show first-use hitches, so they add nothing to the frame_ms data above.

On a 32 GB machine, the opt-in is the difference between paging about 22 GiB and about 1 GiB. That supports keeping it as the stopgap until the cap/eviction change lands.

Nit (optional): APS5_LAZY_SHADER_PREPARE=0 also turns the mode on. ShaderRegistry.cpp:1522 tests std::getenv(...) != nullptr, the same as APS5_SYNC_SHADER_PREPARE at :1073. docs/dev/ARCHITECTURE.md:103 documents =1, and APS5_NO_LIBRARY_PREBUILD (:874) compares against "1".

GPUs tested: NVIDIA GeForce RTX 3060, driver 591.86, Windows Server 2025.
AI-assisted: yes

[adriadam10]

Re-checked. LazyRegistration now asserts the snapshot is deferred with no entries after registration and prepared after the first dispatch; with the lazy branch in ShaderRegistry.cpp disabled, agc_driver_prepared_shaders_lazy and _lazy_sync fail. Built merged onto main: all 11 agc_driver_prepared_shaders* pass; the only other failures (agc_driver_tessellation, agc_driver_preparation_scope_settle) also fail on main.

Memory numbers are backed by @AnthonyClemens (RE2) and @fdfawef324 (Demon's Souls), and the frame_ms data answers the first-use question. Non-blocking: APS5_LAZY_SHADER_PREPARE=0 also enables the mode, same as APS5_SYNC_SHADER_PREPARE. Merging as the opt-in stopgap.

[adriadam10]

Re-checked after main changed libSceAgcDriver/CMakeLists.txt and Driver.hpp: same head, merges cleanly. Built merged onto current main and ran the AgcDriver ctest scope: all 11 agc_driver_prepared_shaders* pass, including _lazy and _lazy_sync; the only failure is agc_driver_tessellation, which segfaults on main too.

Memory numbers are backed by @AnthonyClemens (RE2) and @fdfawef324 (Demon's Souls). Non-blocking: APS5_LAZY_SHADER_PREPARE=0 also enables the mode, same as APS5_SYNC_SHADER_PREPARE. Merging as the opt-in stopgap.

[adriadam10]

Re-checked after main moved again: same head, merges cleanly. Built merged onto current main and ran the AgcDriver ctest scope: all 11 agc_driver_prepared_shaders* pass, including _lazy and _lazy_sync; the only failure is agc_driver_tessellation, which segfaults on main too.

Memory numbers are backed by @AnthonyClemens (RE2) and @fdfawef324 (Demon's Souls). Non-blocking: APS5_LAZY_SHADER_PREPARE=0 also enables the mode, same as APS5_SYNC_SHADER_PREPARE. Merging as the opt-in stopgap.

[adriadam10]

Re-checked after main moved again: same head, merges cleanly. Built merged onto current main and ran the AgcDriver ctest scope: all 11 agc_driver_prepared_shaders* pass, including _lazy and _lazy_sync; the only failure is agc_driver_tessellation, which segfaults on main too.

Memory numbers are backed by @AnthonyClemens (RE2) and @fdfawef324 (Demon's Souls). Non-blocking: APS5_LAZY_SHADER_PREPARE=0 also enables the mode, same as APS5_SYNC_SHADER_PREPARE. Merging as the opt-in stopgap.