fix(agc): read unregistered graphics programs beside registered shaders

[NaitikJoshiPro]

What

Since 39fdc48, an unregistered (raw) graphics program is read only while the shader registry is empty. Once any shader is registered, every draw of a program outside the registered code throws graphics program does not belong to a compatible registered shader. RegisterShader also throws after the first raw draw anywhere.

Compute dispatches don't have this restriction: they resolve a registered program by its registration and read any other address raw. Some titles need both kinds of program. Homebrew built on the oops-sdk does: its display path (src/agc/agc_display.c) registers its shaders through sceAgcCreateShader, and radeonsi (oops-mesa) submits raw programs for every GL draw. SuperTuxKart (STKT00001, project-oops/oops-apps) stops at its first GL draw with the error above.

Fix. Graphics programs now resolve the way compute does:

• An address inside a registered shader resolves to that registration. A registration of another type still throws, in RegisteredProgram.
• Any other address is read raw.

The concern behind the old RegisterShader check stays covered. A draw keeps a raw snapshot that the draw cache revalidates by reading it raw again, so a registration over the same code would give one program two identities. The code range of every raw program that is drawn is recorded, and RegisterShader throws only when the new shader's code overlaps one of those ranges (a shader was registered over a graphics program that was drawn without registration). Registrations elsewhere are accepted. The range is recorded once per raw snapshot: a flag on the snapshot keeps later decodes of the same program off the lock, so the decode path takes no new lock after a program's first draw. Recorded ranges are merged when they overlap or touch, so the check at registration looks only at the predecessor of the queried end.

Concrete case. agc_driver (testProgramSnapshots) now resolves a raw address while another shader is registered. On current main, ProgramSnapshot throws for that address. The test also checks that a registered program requested as another type is still refused, and checks the range bookkeeping with nested, overlapping and adjacent ranges. agc_driver_prepared_graphics used an address outside every registration as its invalid case; it now registers the draw's programs as compute shaders, which is still refused. agc_driver_prepared_unregistered_graphics now checks that the raw-drawn ranges are recorded exactly, starting at the program and not before it, and that unrelated code doesn't count as drawn raw. Before, it checked that any later registration was refused.

Tested

• Vulkan backend, Linux x86-64 (ubuntu-24.04, GCC 14, lavapipe: llvmpipe, Mesa with LLVM 20) on GitHub Actions on my fork: https://github.com/NaitikJoshiPro/AnyPS5/actions/runs/38661687268. The run builds this change on main (ab9a476; the commits since then touch only the relinker and CI tools) with ANYPS5_REQUIRE_VULKAN=1. agc_driver, agc_driver_prepared_graphics (and its _prebuild_unset, _prebuild_0 and _prebuild_1 variants) and agc_driver_prepared_unregistered_graphics pass: 6 of 6, no skips. The workflow is fork-only and is not part of this PR.
• macOS 27 on an Apple M3, using the native Metal port of this driver at NaitikJoshiPro/AnyPS5-mac with this change applied. The same three tests pass with Metal API and shader validation on.
• Not run: Windows. On main, SuperTuxKart stops earlier than this draw: sceAgcInit(state, version) (fix(agc): return 0 for out-of-range sceAgcInit without table lookup #5163), plus PM4 and register states radeonsi uses that the driver still refuses. So this change on its own unblocks no title.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none on its own (SuperTuxKart STKT00001 homebrew also needs fix(agc): return 0 for out-of-range sceAgcInit without table lookup #5163 and further PM4 acceptance)
• GPUs tested: lavapipe (llvmpipe, Vulkan) on Linux x86-64 in CI; Apple M3 (10-core GPU), Metal, macOS 27.0.1, through the Metal port of the AGC driver
• AI-assisted: yes

🤖 Generated with Claude Code

[CnRJay]

Letting raw programs coexist with registered ones, and narrowing the RegisterShader refusal to overlapping code, is a sensible rule. Two problems before this can be judged:

• It hasn't run on the backend it changes. The only runtime testing is on an out-of-tree Metal port (NaitikJoshiPro/AnyPS5-mac). "Not run: the Vulkan backend" means agc_driver, agc_driver_prepared_unregistered_graphics and agc_driver_prepared_graphics haven't been run on any configuration this repository ships. Please run them on Linux or Windows (CI on your fork is enough) and say so. A SuperTuxKart result on the Metal port is evidence for the port, not for this tree.
• A global mutex in the draw decode. NoteRawGraphicsDraw was a relaxed atomic store. It now takes rawGraphicsMutex and does a std::map insert from DecodeGraphicsPrograms, which runs on every draw-decode miss of an unregistered program. With radeonsi-on-oops, the case this PR is for, every GL program is raw, so every queue worker that decodes serializes on that lock. The range only needs recording once per program: doing it where the program is first read raw (ReadRawShader), or behind a flag on the snapshot, keeps the decode path lock-free as it was. RawGraphicsDrawnIn also walks every recorded range below end. It should use upper_bound and look at the predecessor, as the other range maps in the driver do. It only runs at registration, but it is linear in the number of raw programs.

[NaitikJoshiPro]

Rebased on main and changed:

• Vulkan run. agc_driver, agc_driver_prepared_graphics (with its three _prebuild_* variants) and agc_driver_prepared_unregistered_graphics pass on Linux x86-64 with lavapipe (llvmpipe, LLVM 20, GCC 14, ANYPS5_REQUIRE_VULKAN=1, no skips): https://github.com/NaitikJoshiPro/AnyPS5/actions/runs/38661687268. The first lavapipe run caught a real gap. agc_driver_prepared_graphics used an address outside every registration as its invalid case, and that address is now read raw. The test now registers the draw's programs as compute shaders, which is still refused (PreparedGraphics.cpp). The Metal port already carried that test change, but it was missing from this PR.
• Lock on the decode path. NoteRawGraphicsDraw returns after one acquire load once the snapshot's new rawGraphicsNoted flag is set (ShaderRegistry.hpp, ShaderRegistry.cpp), so the mutex is taken only on a program's first raw draw. I put the flag on the snapshot rather than recording in ReadRawShader. ReadRawShader also serves compute programs and the null pixel program, and neither should block a later registration.
• RawGraphicsDrawnIn. Ranges are merged on insert when they overlap or touch. The lookup is lower_bound(end) plus the predecessor, so it costs O(log n). agc_driver gains testRawGraphicsRanges for nested, overlapping, adjacent and empty ranges (tests/Submit.cpp).

The description's testing section is updated.

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
#1191	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#1747	core/libs/prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
#1896	core/libs/prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
#4650	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#4943	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#5115	core/libs/prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
#5431	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
#5480	core/libs/prx/libSceAgcDriver/tests/Submit.cpp
#5500	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#5609	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#5656	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#5867	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#6188	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp
#6235	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp

[adriadam10]

Checked the diff against main: graphics programs now resolve like compute (registered address → its registration, other type still throws; anything else raw), and RegisterShader refuses only when the new code overlaps a raw-drawn range. The per-snapshot flag keeps the decode path off the mutex after a program's first draw, and the range lookup is lower_bound + predecessor.

Built and ran the libSceAgcDriver scope on the merged tree: agc_driver, agc_driver_prepared_graphics (+ prebuild variants) and agc_driver_prepared_unregistered_graphics pass; without the change the new tests don't build. Only agc_driver_tessellation fails, and it also does on main. Windows not run here; merging after the full Build is green.