perf(shader): emit both registration layouts of a pixel shader from one program

[nikowolf123]

What

On a cold cache, registering a pixel shader builds its front end twice. PrepareRegistered prepares every pixel shader through one ShaderPreparationContext, first with the plan's layout and then again with pushConstantSizeBytes = 0 (ShaderRegistry.cpp:754). The layout is not part of the source key, so both requests share one SourceEntry. But the first variant consumes the entry's front-end program (findOrCompileVariant, Recompiler.cpp:952-953). The second layout therefore runs PrepareResourceProgram again (translation, SSA, folding and the eliminator passes, SrtWalker, ResourceTracker) and then the interface passes, although none of them reads the layout. #2526 shares only decode, CFG and structurization between the two calls.

With this change, the context keeps the program it last emitted from, after the interface passes, and emits the second layout from it:

• compileVariant is split into prepareInterface (ApplyStaticInterface, identity and dead-code removal, ShaderInfoCollector) and emitVariant (binding allocation, SPIR-V emission, the registration word-limit fallback, validation). compileVariant is now the composition of the two and behaves as before. This is the first commit.
• The context holds the source entry and its program. Another layout of the same source at the same code address resets bindingLayoutComplete and emits from that program, which the word-limit fallback already does within one variant (Recompiler.cpp:488-494). The compiled information is copied with a new IrProgram::CompiledInfo() const& instead of being moved out.
• A different source or a different code address builds a program as before. The address check matters because the code address is translated into the program as shaderHash, which loop-guard fault records carry.
• Building another program through the context releases the kept one first. The context lives for one PrepareRegistered call, so no program outlives its registration (the rule of perf(shader): build a shader's front end once on first use #471). A compute pair (full, then partial threads, two sources) never holds two programs at once.
• Disk-cache loads and stores, the emission-failure memo and the variant order are unchanged; perf(agc): build shader pipeline libraries when shaders are registered #4152's library prebuild still uses entries.front(). Calls without a context (Recompile, CaptureResources, PrepareShader without one) take the old path.
• APS5_NO_PERF_LAYOUT_PAIR=1 restores the old path. It is listed in NeutralSwitches, so it does not change disk-cache keys. ResourcePrograms() counts the programs built through a context, for the tests.

This does not fix #4399. There, simonbbby's RTX 3090 profile logs both registration variants of one ps5-opengl pixel shader (5,413 code dwords) at 1,273.6 ms and 1,207.2 ms. That shows the duplicate exists on a real shader and costs about as much as the first variant. But the title's ~2 s wait is lost to pipeline creation (~2.2 s), as MarcZX100 concluded in that issue. By default RegisterShader hands the preparation to the background RegistrationPreparePool. The saving reaches frame time only where something waits for it: a draw or dispatch that needs a shader whose preparation is still pending (AwaitRegisteredPreparation), Driver::ResolveShaderAbi when it prepares an ABI that was not registered (inline, on the calling thread), and APS5_SYNC_SHADER_PREPARE=1.

Open PRs #4339 (wave64 layout choice inside compileVariant), #4048 and #4475 (lines of the emission step) edit nearby lines. The conflicts are textual; whichever merges second needs a rebase. #4271 (IR reclamation around ResourceProgram) and my #4883 (front-end resource plan in the disk cache) are merged, and this branch is rebased onto both. After a plan hit the entry has no program, so the first layout builds it through the context and this change lets the second layout reuse it.

Tested

WSL2 Ubuntu (kernel 6.18), GCC, i9-14900KF, lavapipe, configured like CI (Release, SPIRV-Tools ON):

• Full ctest like CI: 0 of 1004 tests failed; 41 skip on this lavapipe setup (subgroup size 8, no 120 Hz display). A second full run with APS5_NO_PERF_LAYOUT_PAIR=1 set for the whole run, made before the test-only follow-up commits, also had 0 failures and the same skips. The last commit only adds a check to agc_shader_memory_tests; its seven ctests pass on the head. tools/check_test_registration.py passes. tools/check_skips.py flags guest_video_out_high_frame_rate ("current display does not provide 120 Hz"); unmodified main 43e3a70 skips it the same way in the same tree.
• New agc_shader_layout_pair and agc_shader_layout_pair_off (0.3-1.3 s each under load; also run inside agc_shader_memory). A pixel shader with a buffer load is prepared through one context with a 128-byte push range and with none, in both orders and in five configurations: plain modules, registration modules at push offset 0, registration modules at offset 128 (the fragment slot on devices with graphics pipeline libraries), a word limit that forces the fallback to plain modules, and registration modules at offset 128 with two interpolated inputs. Each pair must build one program with reuse and two without.
• Each layout must match a compile of the same code on its own source without a context: SPIR-V, binding layout, push range, UsesPushData, every artifact field except the variant id (fragment parameters element by element), and the compiled shader information. The materialized result must also match an uncached Recompile. The test also checks that the context does not reuse a program across code addresses or across sources at one address, that it releases the kept program when it builds another one, and that CompiledInfo() and TakeCompiledInfo() agree on a compute program with a nonzero user-data base and scratch size.
• Negative controls, each a one-line change with the tests unchanged. In each case agc_shader_layout_pair fails and _off passes. Never reusing the program fails with "the second layout ignored the reuse switch", because it builds 2 programs as main does. The other controls remove the source check (the SPIR-V differs from the isolated compile), the code-address check, the release before another build, or the bindingLayoutComplete reset ("binding layout already allocated"), empty the bindings of the copied compiled information, or change the barycentric flag, a fragment parameter or the runtime ABI version of the reused artifact. Zeroing the user-data base, scratch size and export mask of the copy fails both tests, since that check does not depend on the switch.
• agc_shader_memory_tests builds and passes on the refactor commit and on the perf commit alone.

Performance: I have no PS5 and no game files, so there is no trace from the Performance report form. The closest equivalent without a title is a synthetic measurement:

• A harness (outside the repository; I can attach it) registers one synthetic wave32 pixel shader the way PrepareRegistered does: one context, PrepareShader with a 128-byte push range (A), then with none (B). The program is a chain of buffer loads, f32 ALU and exec-masked branches.
• Each run is a fresh process with ANYPS5_NO_SHADER_CACHE=1, so the in-memory and disk caches are cold. The build is Release with SPIRV-Tools OFF, as release.yml builds.
• The timer also covers destroying the context (teardown). On main each PrepareShader frees its own program inside A and B. On the branch the one program is freed with the context, so pair = A + B + teardown is the comparable total.
• main 43e3a70 and this branch were built in the same tree and run alternately, 10 rounds (5 in each order), on the same machine in one session. Other builds moved the load average between 2 and 23 (median 6), so absolute times are noisy.

Medians of each column, ms:

guest dwords	registration modules	main A / B / teardown / pair	branch A / B / teardown / pair	B	pair
2,005	no	81.9 / 74.7 / 0.1 / 155.6	76.8 / 13.4 / 5.2 / 95.3	-82%	-39%
2,005	yes	93.5 / 85.1 / 0.1 / 178.3	87.6 / 28.0 / 5.2 / 121.3	-67%	-32%
8,005	no	608.5 / 576.6 / 0.2 / 1,182.7	585.8 / 120.0 / 28.0 / 734.1	-79%	-38%
8,005	yes	747.5 / 715.1 / 0.2 / 1,469.6	732.8 / 239.0 / 28.6 / 1,002.7	-67%	-32%
16,005	no	1,417.3 / 1,316.2 / 0.5 / 2,733.1	1,348.9 / 311.4 / 61.9 / 1,720.5	-76%	-37%
16,005	yes	1,724.1 / 1,602.1 / 0.4 / 3,343.1	1,668.4 / 631.0 / 62.0 / 2,377.2	-61%	-29%

A is a few percent lower on the branch only because main frees the first program inside A, while the branch frees it in the teardown column. B on main also includes freeing its program and B on the branch does not, so the pair column is the fair comparison.

In every run, the SPIR-V and binding layout of both layouts hash the same on main and on the branch. With registration modules these synthetic modules exceed the 200,000-word registration limit, so each variant is emitted twice (module, then fallback), and that emission is most of what remains of B. How much a title gains depends on how much of its cold pixel-shader registration is front end; I have not measured a title. I can attach the harness source, the scripts that build and run it, and the raw JSONL (one line per process, with timings and output hashes). An earlier batch stopped the timer before the context was destroyed and reported 2-3 points more pair saving; this batch supersedes it.

Native Windows 11 (WinLibs GCC 15.2.0, Release with tests) on an RTX 5080 (NVIDIA 616.92), same head: full ctest with ANYPS5_REQUIRE_VULKAN=1, 1021 tests, 0 failed, 7 skipped as tools/check_skips.py --job windows expects; agc_shader_layout_pair and agc_shader_layout_pair_off pass. Not tested with a title.

• Rebased onto main 0c3dc9c for this pull request: it builds in the CI configuration and the 34 shader, pipeline-library, structurizer and control-flow tests (ctest -R 'agc_shader|agc_driver_prepared|agc_pipeline_library|agc_structurizer|control_flow') pass on lavapipe.
• Rebased on 2026-10-14 onto main 0b6d033 (head 7028eda is now 49de394). One conflict, in Recompiler.cpp: perf(recompiler): keep the front-end resource plan in the shader disk cache #4883 added FrontEndBuilds() where this change adds ShaderPreparationContext::ResourcePrograms(), and both are kept. The rest applied cleanly, and the diff against main has the same added and removed lines as before (7 files, +239/-18).
  Since 0c3dc9c, perf(shader): reclaim unreachable cached source IR values #4271 makes ResourceProgram clear block register state and reclaim unused values (PrepareShader passes reclaim = false), and perf(recompiler): keep the front-end resource plan in the shader disk cache #4883 loads a stored resource plan in getSource before the front end runs. Neither changes which program the context keeps or when it releases it. After a stored-plan hit the entry has no program, so the first layout builds one through the context and the second reuses it. I checked this with a cache directory that held only the stored plans of the --layout-pair fixtures: each pair built one program, or two with APS5_NO_PERF_LAYOUT_PAIR=1, and both layouts matched their isolated compiles. The fixture's later checks assume no disk cache, as ctest runs it. The stored formats are unchanged, so FormatVersion stays 30. fix(shader): verify materialized snapshot identity before memo reuse #4757 was closed unmerged, so there is nothing of it to keep.
  Tests on 49de394, WSL2 lavapipe configured like CI: the full ctest had 0 failures out of 1112 tests, with 43 skipped; tools/check_skips.py flags only guest_video_out_high_frame_rate (no 120 Hz display), as before. All 29 tests of ctest -R 'agc_shader|shader|recompiler' pass, including the two layout-pair tests, agc_shader_memory, agc_shader_disk_cache and both front-end pair tests. agc_shader_memory_tests also builds and passes on the refactor commit and on the perf commit alone. The branch also merges cleanly with the newer main 107190d, and that merge passes the full ctest on the same setup (0 failures out of 1114 tests, 43 skipped, the same check_skips.py result).
  Native Windows 11 (WinLibs GCC 15.2.0, Release with tests) on an RTX 5080 (NVIDIA 616.92), with ANYPS5_REQUIRE_VULKAN=1: all 29 tests of the same filter pass. The full ctest had 1 failure out of 1135 tests, with 7 skipped as check_skips.py --job windows expects. The failure is guest_avplayer ("enable video failed"), which fails intermittently in this tree (10 of 24 runs). Its executable is built only from libSceAvPlayer (which compiles in libkernel's TimedWait.cpp), libc and the prebuilt FFmpeg, none of which this branch changes, and main's Windows CI passes it.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• GPUs tested: lavapipe (Mesa 26.0.8, llvmpipe) on WSL2 Ubuntu 26.04; NVIDIA GeForce RTX 5080, driver 616.92, Windows 11
• AI-assisted: yes. Written with Claude Code; I reviewed every line and ran the tests above.

🤖 Generated with Claude Code

[Ondra97]

I reviewed #4884 against #4757. Your retained interface program changes variant preparation; #4757 fixes a reproduced CPU collision in materialized-result memo reuse. The direct overlap is the shared test insertion points and Windows CMake test-property list. I see no duplicated implementation of the memo identity fix.

I propose landing #4757 first, then rebasing #4884 while retaining the owned snapshot and exact Matches checks at both lookups. Both layout-pair CTests and agc_shader_snapshot_memo should remain registered, and the collision case should be checked with layout reuse enabled and with APS5_NO_PERF_LAYOUT_PAIR=1. I have not tested the combined branches. Would you agree to this order?

AI-assisted: yes. — Codex

[nikowolf123]

Agreed: #4757 first. Once it merges I'll rebase #4884 onto it, keep the owned snapshot and the exact Matches checks at both lookups, keep agc_shader_layout_pair, agc_shader_layout_pair_off and agc_shader_snapshot_memo registered, and run the collision case with layout reuse on and with APS5_NO_PERF_LAYOUT_PAIR=1 before pushing.

AI-assisted: yes.

[adriadam10]

Checked on Linux (PR merged onto current main, scoped build + ctest of libSceAgc/libSceAgcDriver): agc_shader_layout_pair, agc_shader_layout_pair_off and agc_shader_memory pass; without the change the tests don't build (ResourcePrograms()). The reuse is keyed on the source entry and the code address, the kept program is released when another is built, and bindingLayoutComplete is reset the same way the word-limit fallback already does. Not verified here: the gain on a real title (the numbers are synthetic, as the description says). Merging once the Windows build is green; #4757 will need a rebase on top.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#5466	core/shader/recompiler/Recompiler.cpp L467-468
#5473	core/libs/prx/libSceAgcDriver/CMakeLists.txt L1313