perf(shader): record a specialized buffer atomic's fault without the address tables

[oneandonlydean]

What

A program with an atomic on a V# it specializes is compiled as address-based: since 7431e5a TakeGpuDescriptor sets usesDma for it, so that the atomic's unaligned-address check can call the fault function. The atomic itself goes through the element's storage buffer, and nothing in such a program reads the address tables.

Being address-based costs in the driver. The build leases all registered guest memory (AcquireRegistered), and a V# inside it is served by the leased range and bound in place inside a host import. None of the program's elements reaches the staging decision in UploadPrepare: not the atomic element, and not a read and written element the dispatch sweeps (#2681). In one title's particle update a 20-byte atomic counter kept a 10 MiB pool in place, every load a PCIe round trip.

The fault function has had a form without the tables since #759 (usesFaultBuffer). Now:

• TakeGpuDescriptor sets usesFaultBuffer instead of usesDma for an atomic on a specialized V#;
• EmitRuntimeBufferAtomic records the fault through RecordBdaFaultWords, which has both forms, and stops the invocation as before;
• the 64-bit fault address needs shaderInt64, checked as in EmitRuntimeScalarBufferLoad;
• a tessellation-control program with such an atomic is refused with the message it got before through ValidateBdaTarget.

The fault record (reason, address, bytes, pc), the atomic and the in-bounds checks are unchanged. A program that is address-based for another reason is compiled as before. No driver change: a program with a fault buffer and no tables already takes the descriptor path.

Tested

Linux (CachyOS, kernel 7.2, GCC 16.2, RTX 4090, NVIDIA 610.57.04), Release, BUILD_TESTING=ON, on main 2a55cfe.

• agc_shader_memory, new case verifySpecializedBufferAtomicBindings: a buffer_atomic_inc on a specialized V# must give no address table binding, one fault buffer, one call of record_bda_fault, logical addressing and the element's atomic flag; without shaderInt64 the program is refused. On main it fails with specialized buffer atomic: a buffer_atomic_inc through a V# the program specializes binds the address tables.
• ctest -R "agc|shader": 456 of 456 on main and on this branch.
• python3 tools/check_conventions.py --base origin/main passes.
• A 2D title we test with: runs to gameplay as on main, no skipped draws.
• A 3D title we test with does not run on main yet (it stops at a storage image over a depth surface, feat(libSceAgcDriver): storage image access to depth surfaces #4500). Measured on a local tree, main 682ef0c plus the pull requests the title needed that day, with and without this change, runs alternated, caches warm:

per present	without	with
the particle update kernel, GPU	5.2 to 6.7 ms	0.06 to 0.07 ms
staging copies in + out, title	0.5 ms, 5 MiB each way	1.4 ms, 15 MiB each way
GPU busy, title	28.5 to 29.2 ms	23.1 to 24.2 ms

scene	without (5 runs)	with (6 runs)
title, fps	24.5, 24.0, 23.8, 24.2, 24.6	25.35, 25.7, 25.45, 24.7, 25.55, 25.45
menu, fps	22.1, 21.4, 21.9, 22.0, 22.0	24.4, 24.75, 24.4, 24.0, 24.75, 24.25
first level, ms per frame	36.6, 37.0, 37.8, 37.1, 36.6	33.6, 33.4, 33.8, 33.6, 33.6, 33.2
a crowd scene, ms per frame	76.7, 78.0, 77.3, 77.5, 76.8	78.5, 75.0, 78.2, 76.7, 75.7, 75.7

The per-program times are from APS5_PROFILE_GPU=1 (one run each side); the scene numbers from APS5_PROFILE_DRAW=1. Other work ran on the machine between and during some runs; a sixth run without the change was dropped for that. The checked frames of the title, menu and level look the same and get the same reference scores with and without the change.

Not tested:

• Summary lines of APS5_TRACE_FRAMES: the 3D title does not run on main, and the local tree is older than the trace.
• The unaligned atomic on a device. The test checks the module (bindings, capabilities, the call of the fault function), not a fault read back from a GPU.
• Windows.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• AI-assisted: yes (Claude Code)

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#4279	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L2279