fix(agc): address-based builds map only memory mapped for the GPU

[Raghav314]

What

An address-based build put every readable registered range in its BDA table and imported or mirrored it, including memory the guest mapped for the CPU only. On the console a mapping reaches the GPU only if its protection has GPU access (0x10 read, 0x20 write).

• The guest allocation registry records whether a range has GPU access. libkernel sets it from the protection at sceKernelMapDirectMemory, flexible maps, fixed remaps and sceKernelMprotect. Reservations and the main image have none; a title's mprotect with GPU bits gives the image pieces access.
• An address-based build leaves out ranges without GPU access. Nothing else changes: stores are still decided by CPU write access and in-place import, and AnyPS5's own libc heap is still registered with GPU access.
• GuestAllocations::Mutation::Add and Protect take the flag.

Sources:

• The protection bits: shadPS4 src/core/memory.h MemoryProt (CpuRead = 1, CpuWrite = 2, GpuRead = 16, GpuWrite = 32).
• GPU page table entries exist only for ranges with GPU access: KytyPS5 src/kernel/memory.cpp CountPageTableEntries(bool gpu) counts a range for the GPU only if its protection has PROT_GPU_READ or PROT_GPU_WRITE, and sceKernelGetPageTableStats reports the CPU and GPU counts separately.
• PPSA36082 maps memory the CPU uses with 3 and sets the GPU bits where the GPU reads: its bundled libc.prx maps the malloc heap with sceKernelMapNamedFlexibleMemoryInternal (4h6F1LLbTiw), protection 3, at both call sites (+0xb2eb, +0xb4fd); its executable's segments carry only R/W/X flags, and it mprotects the two pieces the GPU reads with 0x12 and 0x11.

PPSA36082

During loading and the main menu it registers: the 1 GiB malloc heap at 0x200000000 with 3, 6784 MiB of direct memory with GPU read and write, and the two 32 KiB image pieces. Its first address-based draw runs on the queue the loader's fence waits on. That draw imported the heap; the import fails to pin (the kernel logs Cannot map memory with base addr 0x200000000 and size of 0x40000 pages), the fence is late, and the 512 MiB upload pool overflows (eboot 0x28fba9). With this change the heap is not in the table and nothing fails to pin.

In my local build I bypassed both problems by hand: a driver option, APS5_GPU_EXCLUDE=0x200000000+0x40000000, kept that address range out of address-based builds, and a patched executable got through loading with a 2.5 GiB upload pool instead of 512 MiB. Both are specific to this title. This change replaces the first: the title maps its heap with 3, which has no GPU access, so address-based builds leave it out without an option. Loading with the original 512 MiB pool also needs the changes below.

It also needs #861, #1367, #1249, #290 and a follow-up I will open after this one, which imports GPU-mapped memory when it is mapped and depends on this PR. With those, on main df16c4c with an earlier version of the follow-up, 40 s runs to the main menu: 4/5 pass with this change (the other overflowed at 4.1 s; the pool's margin is thin and that is separate work), 0/3 without it (each overflowed at 503 MiB with the pin failure above).

Local changes not in this PR

I also have these on local branches. They finish making the GPU follow the guest's protection, but I am not opening them because no title I can run needs them. Do you want PRs for them?

• GPU read and write kept apart: the registry records GPU read and GPU write separately; the BDA table grants reads only with GPU read and stores only with GPU write (today CPU write decides stores); AnyPS5's own libc heap is registered without GPU access, as the console's libc.prx maps its heap with 3. PPSA36082's only range where CPU and GPU write differ (an image piece mprotected with 0x12) gets no GPU stores, and it allocates nothing through AnyPS5's libc heap.
• GPU access on every path, not just address-based builds: buffer descriptors read zeros from memory without GPU read and drop stores to memory without GPU write (memory with GPU write only is copied in so stores are exact); the command processor (command buffers, WRITE_DATA, DMA_DATA, labels, waits, fills), render targets, index and vertex inputs, DCC keys, display buffers and shader code fault on such memory; texture reads see zeros. The driver's GDS backing and AvPlayer's fallback frames are mapped with GPU access. A counted PPSA36082 run made no access to memory without GPU access.
• Mappings with GPU access but no CPU read (0x10, 0x30) reach address-based builds, and AMM PRT pages with no physical memory read zero and drop GPU stores, through an unbound Vulkan sparse buffer (strict residency). Sources: amdgpu AMDGPU_PTE_PRT, which turns off faults on unmapped PRT accesses (gmc_v8_0_set_prt), and RADV reporting sparseResidencyNonResidentStrict. PPSA36082 makes no such mappings and imports no sceAmpr* function.
• The main image is registered when the process starts on both OSes. Today Linux registers it at any mprotect and Windows only at an mprotect of the image. PPSA36082 mprotects its image, so it is registered either way.

Tested

Linux (CachyOS), x86-64, GCC, 7800X3D / RTX 4090.

• guest_memory: a direct map with 3 has no GPU access, 0x32 has; sceKernelMprotect to 0x13 adds it and back to 3 removes it; flexible maps with 3 and 0x33 likewise; the main image has none. Mapping with the GPU bits ignored fails it, and so does registering the image with GPU access.
• agc_driver_graphics: an address-based build includes a range with GPU access and leaves out one without. Dropping the check fails it.
• Test blocks that GPU work reads are registered with GPU access.
• ctest: 494/494.
• Windows: cross-built with MinGW-w64 GCC 15.2.0 and run under Wine 11.19. agc_driver_graphics and agc_driver_flat_store pass; guest_memory passes the new checks, then stops at CheckMlock (the working-set query), as it does without this change.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none (PPSA36082 also needs fix(guest-arena): place guest address space in the PS5 window on Linux #861, fix(videoout): create the window and the device when VideoOut starts #1367, fix(relinker): write a bucketed hash table for Linux guest modules #1249, feat(agc): host import windows, least-recent eviction under the budget #290 and the follow-up above)
• AI-assisted: yes

[MarcZX10]

Reviewed the complete diff against current main f145139. The registry now carries GPU-access state through direct/flexible mappings and protection changes, and address-based builds omit CPU-only registrations. The tests cover map and mprotect transitions plus BDA inclusion/exclusion; the Linux and Windows CI runs and Conventions all pass. The PR description clearly scopes the remaining directional-permission and CPU-unreadable cases as follow-up work. I found no blocking issue in this change.

[popcoder565]

Native Windows run, since the description only covers Windows under Wine.

Setup: Windows 10 (19045), RTX 3060 Laptop (NVIDIA 616.92), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release. Head cf6ae853 merged onto main d9484c7.

• Full ctest: 494/494, twice, with the GPU tests on the RTX 3060.
• guest_memory, agc_driver_graphics and agc_driver_flat_store: 5/5 each. Natively, guest_memory runs past CheckMlock, where it stopped under Wine.

No title was run.

🤖 Addressed by Claude Code

[TheAggie]

A data point for the follow-up that registers AnyPS5's libc heap without GPU access: vkQuake (PPSA99010, homebrew, through RADV) allocates ~10,000 small blocks through AnyPS5's libc heap, and every address-based build refreshes a heap mirror for each. Leaving only the libc-heap blocks out of address-based builds in a local spike took the demo from ~22 to ~44 FPS, with correct rendering and no BDA faults. RADV's own memory is also freeable, so excluding every releasable range broke rendering; a real GPU-access flag, as in your follow-up, is what's needed.

Windows 11, RTX 5070 Ti. AI-assisted (Claude Code).

[oneandonlydean]

Conflicts with main in core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp; please rebase.

[Raghav314]

Rebased on main c9da0f9. The conflict in BdaResources.cpp was two new test calls in the same place; both are kept.

Main's newer tests that register memory the GPU reads (BdaResources.cpp import-crossing and heap-mirror tests, BdaSpanReads, FillCoverKeys, PixelUint16Export, SampledCacheBudget, StorageCacheBudget, and one in Recorder.cpp) now register it as GPU-mapped, like the older ones this PR already changed. Without that, address-based builds would leave their memory out.

The test registration step fails on main itself right now: e5c6099 runs agc_driver_recorder_tests in ctest (agc_pipeline_cache) but it is still in KNOWN_UNRUN in tools/check_test_registration.py, so CI will stop there until that line is removed on main.

[Raghav314]

The write-watch (macos-15-intel) failure is a write_watch_threads timeout in code this PR doesn't touch (it passed on the previous run, 38037405404); could someone re-run it?

[adriadam10]

Checked the diff against main: every Add/Protect caller passes the GPU bit from the protection (0x30), reservations and the main image get none, and address-based builds skip ranges without it. Build and ctest of libs, AgcDriver, libc and libkernel pass (447/447, incl. guest_memory and the agc_driver_* tests). Also confirmed by @popcoder565 (native Windows, 494/494) and @MarcZX100's review. Merging once CI is green.