perf(agc): return freed memory when shader preparation drains

[milescoviello]

What

Ghost of Tsushima (PPSA02225) ends its first run, with no resource plans in the shader cache, at about 13.3 GB resident, against about 9.0 GB once the cache holds them. The game is at the same point in both runs (the same 6.5 GB of guest memory resident), so the difference is host memory.

It is memory the registration prepare pool has already freed. On a first run the pool prepares the title's shaders on its threads, and each thread allocates from its own glibc arena. /proc/<pid>/smaps near the end of a first run shows 5.7 GB of anonymous memory in those arenas, against 1.9 GB in a warm run. A malloc_info taken at 110 s from inside the process counts 4.6 GB of the 6.5 GB the arenas hold as free, and a malloc_trim(0) at that point takes the process from 13.6 GB to 9.5 GB resident. glibc returns freed memory from the middle of an arena only on malloc_trim, so without it the memory stays resident for the rest of the run.

When the pool has no task left, the worker that finished last now calls malloc_trim(0), at most once a second. That gives the free pages of every arena back to the system. Only glibc builds call it (__GLIBC__).

The peak is unchanged: the first preparation burst does not drain the pool until it ends, so the arenas still reach their size before the first trim. I also tried trimming every 2 s while the pool is busy. That lowered the peak by about 1 GB (13.2 to 12.4 GB), but it took 2.8 s of trimming in a 150 s run with single trims up to 202 ms, so it is not part of this PR.

Tested

Ubuntu 24.04, GCC 13.3 (CI's compiler), glibc 2.39, Release with CI's configure flags. Measured on main 31bfc4e, then rebased onto main daf256b:

• Ghost of Tsushima (PPSA02225), relinked with --to-intel and run on Linux (RTX 4070 Ti, NVIDIA 580.142) for 150 s with a warm Vulkan pipeline cache but no resource plans. The runs differ only in libSceAgcDriver.prx; resident memory sampled every second:

  first run	at the end	peak
  main	13,405 MB	13,571 MB
  this PR, with the timing prints below	9,245 MB	13,305 MB

  A warm main run ends at 8,946 MB. After the rebase, a first run with this PR ends at 8,960 MB (peak 13,008 MB).

• With temporary timing prints (not part of this PR), a 150 s first run trimmed 5 times, 456 ms in all, the longest trim 243 ms. Each trim runs on the worker that drained the pool, and holds each arena's lock while it trims that arena, so a thread that allocates from the same arena meanwhile waits. Without the one-second limit the pool trimmed 37 times in such a run.

• Some first runs on this machine stop with a glibc heap check (corrupted size vs. prev_size, free(): invalid next size (fast), double free or corruption (!prev)), with and without this change: 3 of 22 on main and 4 of 28 with this PR, and in 19 interleaved pairs 3 on main and 1 with this PR. The rate is the same with and without this change, and they look like this machine's memory rather than code: memtester inside the VM reports a word whose bit 6 reads wrong (0xffffff5fffffffbf != 0xffffff5fffffffff), and in each of the four cores I examined the damaged heap word differs from its right value in that same bit.

• All 514 tests in libSceAgcDriver pass on main daf256b. agc_driver_stencil_ref_export is skipped because the device lacks VK_EXT_shader_stencil_export, and agc_host_import_sparse and agc_driver_sparse_dma_buf_import are skipped because /dev/udmabuf is not accessible here.

• A MinGW-w64 GCC 13 -fsyntax-only compile of ShaderRegistry.cpp has no errors.

• tools/check_conventions.py --base upstream/main passes. tools/pr_overlap.py --branch reports fix(agc): wait for queued registration preparations before accepting a submission #5007, which changes the same worker loop to record finished tasks and to notify idle after every task. The two conflict only in text: whichever merges second keeps its own drained check for the trim.

Not checked on Windows (MinGW does not define __GLIBC__, so nothing changes there), on another Linux libc or on a console.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• AI-assisted: yes

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#5007	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp L14, L925-926

[adriadam10]

Checked the diff (one malloc_trim(0) call, glibc-only, limited to once a second, run after idle is notified so waiters aren't delayed), the build and the libSceAgcDriver test subset on the PR merged onto main. The only failure, agc_driver_tessellation, fails identically with the PR's change reverted, so it is not caused by this PR. The end-of-run RSS drop (13.4 GB → 9.0 GB) is the author's measurement; merging once the full Build is green.