fix(agc): defer unrouted vertex barriers

[BasilF-dev]

What

• Defer registration-time preparation of type-2 vertex programs that contain s_barrier while VGT_SHADER_STAGES_EN routing is still unset.
• Let graphics ABI resolution prepare the program once the real draw routing is known, so GS_FAST_LAUNCH can select the mesh/workgroup path instead of caching a premature Vertex-stage translation failure.
• Keep true Vertex-stage barrier/LDS programs unsupported: once routing resolves to a normal vertex path, the existing exact failure remains.

This progresses Marvel's Wolverine (PPSA03671), Game bring-up #5516. That report has 131 registered shaders failing with:

stage=1 opcode=Barrier reason=s_barrier in a vertex program that accesses LDS or GDS or writes memory is not implemented

The regression registers a type-2 shader with an LDS write plus s_barrier, then resolves it with the Wolverine GS_FAST_LAUNCH state already supported by #4702 (VGT_SHADER_STAGES_EN=0x482030, point list, GE_CNTL=0x201). On unmodified main, registration prepares it as a Vertex shader and hits the barrier error. With this change, registration defers until draw routing is available and the same program resolves through the mesh/workgroup path.

Tested

Linux Mint 22.3, x86_64, NVIDIA GeForce RTX 2050, driver 595.99.02.

• git diff HEAD^ --check — passed
• python3 tools/check_conventions.py --base origin/main — passed
• agc_driver — passed
• agc_driver_graphics — passed
• agc_driver_mesh — passed on GPU
• agc_driver_prepared_shaders_registered_workgroup — passed
• agc_driver_prepared_shaders_registered_workgroup_sync — passed
• agc_vertex_barrier — passed
• python3 tools/pr_overlap.py --base main --branch basil/wolverine-barrier --repo boykopovar/AnyPS5 — no overlap with open pull requests
• full clean build — passed
• full ctest --test-dir build-wolverine-final --output-on-failure -j2 — 1216/1216 passed, 12 skipped

PPSA03671 is not present on this machine, so the title itself was not run locally. The title-side failure is from Game bring-up #5516; the regression reproduces the registration-to-fast-launch path with the same routing state.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA03671 Marvel's Wolverine (PPSA03671) #5516
• GPUs tested: NVIDIA GeForce RTX 2050, driver 595.99.02, Linux Mint 22.3
• AI-assisted: yes (ChatGPT)

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6612	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp L903-904

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 5115	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 6134	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 6188	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 6867	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 6977	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp

[nicolasfalesy]

Tested on Linux (Arch), c++ (GCC) 16.2.1 20260810, Release, BUILD_TESTING=ON, main at 5d9dcc42d2ec5401aebd089028b8d4d7410a77b4.

This PR does not compile on current main. Merged onto main, libSceAgcDriver fails (2 of 2 builds):

core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp:837:80: error: 'struct AgcDriver::DriverDetail::{anonymous}::RegisteredPreparation' has no member named 'decoded'

ShaderRegistry.cpp alone compiles on main (2 of 2). With this PR merged it fails (2 of 2).

Most likely cause: the PR is based on 4bdfc9c, before 54f5c41 ("perf(agc): free a registered shader's decoded program once it is planned") moved the decoded program out of RegisteredPreparation into the local decoded in PlanRegistered. The new unroutedBarrier check at ShaderRegistry.cpp:804 of this PR still reads plan->decoded.instructions.

Suggested fix: rebase and read decoded.instructions there. I checked that this one-line change makes ShaderRegistry.cpp compile on the merge.

AI-assisted: yes

[nicolasfalesy]

Re-tested on Linux (Arch), c++ (GCC) 16.2.1 20260810, Release, BUILD_TESTING=ON: head ded05f528 merged onto main at 47a266cf9.

The compile error from my earlier report is gone. The merge is clean, and libSceAgcDriver (including ShaderRegistry.cpp) builds.

AGC tests (ctest -R agc, 656 tests, 2 runs): 655 pass, 3 skip, 1 fails in both runs. agc_vertex_barrier passes. The one failure is agc_driver_sample_multisample_target ("storage image multisampling is unavailable on the target device"). It fails the same way on main alone on this GPU (Intel Arc, Mesa ANV), so it is not caused by this PR.

AI-assisted: yes

[Neithennn]

Windows NVIDIA run, since the description lists Linux only.

Windows 11 Pro 26200, NVIDIA GeForce RTX 4080 (driver 617.42), AMD Ryzen 7 5800X, WinLibs GCC 15.2.0 r7, Ninja, Release, bundled dependencies, BUILD_TESTING=ON, ANYPS5_NO_SHADER_CACHE=1. Head ded05f528 merged onto main f51724078; full build succeeds.

• agc_driver, agc_driver_graphics, agc_driver_mesh, agc_driver_prepared_shaders_registered_workgroup, agc_driver_prepared_shaders_registered_workgroup_sync, agc_vertex_barrier: pass, with ANYPS5_REQUIRE_VULKAN=1.
• Full ctest -j 8: 1398/1398. Unchanged main on the same machine: 1398/1398.

No title run.

AI-assisted: yes (runs done with Claude Code).