fix(agc): place device memory in a host heap only when that heap has room

[fdfawef324]

What

When a device-local allocation would exceed its reported heap budget, DeviceMemoryType currently selects the first mask-allowed host type without checking that host heap's budget. This change selects a host type only when its heap has room for the allocation. It shares the existing coherent-buffer fit rule through Context::HeapFits, searches later allowed host heaps when an earlier one is full, and keeps the device-local type when no host heap fits.

Placement diagnostics include the selected host heap's usage and budget. A separate rate-limited line records keeping the device-local type when allowed host types exist but none fit. A mask allowing no host type remains silent.

Rebased onto fc623324a after #5987 merged. This PR now contains one guard commit, preserving main's AllocateImageMemory routing, buffer-pool trim, texture-cache reclaim and eventual host fallback after an actual device allocation refusal. It does not restore the superseded AllocateDeviceMemory helper. Thanks to @blawesom for the offered 321ab70c9 rebase; the author and earlier Claude credit are retained, with credit for that recovery work.

This is a placement decision, not a hard memory cap. Main's existing refusal path can still allocate in host memory after the device driver refuses a device-local allocation, even when the host budget reports no room. Reports are snapshots and can lag or be shared by concurrent allocations. On Windows, WDDM can also move device-local allocations into system memory. A device buffer kept device-local may remain in the existing bounded pool after release; #4472 separately changes retention under pressure.

Demon's Souls evidence remains a hypothesis. A historical Windows run on a5e2a80c6 with #5870, #6280 and #6412 died about ten seconds after a Windows event 2004 reported 95.65 GiB committed against a 95.9 GiB limit. The app had 32.5 GiB private, and about 45.3 GiB of commit belonged to no process. There was no failing-allocation stack showing this placement path. This PR neither identifies all that memory nor establishes a title memory or FPS improvement.

@parl0v's RTX 4060/Windows report identifies a host-type allocation refusal after the host heap filled. Their later on/off A/B peaked below the host budget and never logged the guard; both arms survived 3/3. It supports the failure mode, not proof that this guard prevented the failure. No claim is made to fix #5961's retained storage-image cause.

Nearby work: #4472 moves placement counters and changes pool retention; #7119 adds failure diagnostics. Their current heads already conflict with fc623324a, including shared buffer-pool tests, and need manual composition. Preserve the host-room guard with #4472's counters and #7119's diagnostics. Proposed order is this guard first; coordination is pending on #4472 and #7119. No combined-build result or merge-order agreement is claimed here.

Tested

Exact candidate: 4e8edc80e2646ad7619ffafb3bbb24c3f08949c6, based on fc623324a2ab3443cb69a6d0199141b8b2fee604, Windows Server 2025, MinGW GCC 15.2 Release/O3. Independent source review accepted the three-file change; the current fallback and multisample allocation paths remain unchanged.

New CPU/mock coverage checks host room, exact fit, nearly full/over-budget/zero budgets, a second host heap and its type mask. Integration fixtures call the actual DeviceBuffer and AllocateImageMemory paths: when neither heap fits, success stays device-local; a mocked refusal attempts device-local before host memory.

An isolated control copies the exact header and DeviceMemoryFallback.cpp. With only the host-room lookup changed back to the old host lookup, all six budget/mask assertions fail; the unchanged source passes. Fresh O3 control executables plus both test runs took 2.81 seconds of bounded child-job time. This isolated fixture uses native Windows exception handling and no driver library or GPU; it is separate from the normal project targets and does not measure game speed.

The normal Release targets built successfully. The cold build reached its 900-second bound without a compiler error; the remaining 25 incremental steps completed in 86.13 seconds. All five selected CPU/mock CTests passed in 11.58 seconds with no skipped cases: agc_device_memory_fallback, agc_buffer_pool, its two small-device variants and agc_driver_graphics. Conventions passed against the pinned base. The source remained clean throughout; generated TraceRevision is 4e8edc80e264, dirty 0.

Fresh current-head managed-slot Vulkan batch: agc_driver_pooled_buffer_range, agc_driver_sampled_cache_budget, agc_driver_sampled_cache_budget_memory and agc_driver_storage_cache_budget, 4/4 passed in 9.36 seconds, no skipped cases, with ANYPS5_REQUIRE_VULKAN=1. These are real Vulkan adjacency tests, not a real both-heaps-full reproduction. Actual executables and runtime libraries were hash-checked before and after. A clean merge into main 64b57d1b0 was checked separately; all tests ran on the exact candidate.

Historical evidence: @blawesom reported 1338/1339 tests on RX 9070 XT/RADV Linux for their offered 321ab70c9, with the stated main tessellation failure. That is their external result on the offered head, not a local run or a test of this current head. The original 56d44ed2c branch had seven selected cases pass on our NVIDIA machine; those are historical head-specific results, not fresh verification.

Not tested: a title on this rebased head; a real device with both heaps unable to fit; Linux locally; full suite. No title memory decrease or FPS gain is claimed.

Checklist

• Based on current main; no other open PR implements the same functions (host-room guard; nearby perf(agc): stop pooling released device buffers over the memory budget #4472/chore(agc): report heap budgets on buffer allocation failure #7119 composition described above)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source (none added)
• Generic behaviour, not specific to one title
• Depends on: none (fix(agc): place device memory in host memory when the driver refuses a device-local allocation #5987 merged)
• Fixes: none
• Unblocks: none (the Demon's Souls context is an unproven placement hypothesis)
• GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100 (the four current-head Vulkan adjacency cases above)
• AI-assisted: yes (Claude Code and Codex)

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Co-Authored-By: blawesom blaplane@gmail.com
Co-Authored-By: Codex noreply@openai.com

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 4072	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 4472	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp core/libs/prx/libSceAgcDriver/tests/BufferPool.cpp
PR 6683	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 6887	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 6965	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 6973	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 6987	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 6989	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 6995	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 7002	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 7119	core/libs/prx/libSceAgcDriver/tests/BufferPool.cpp
PR 7332	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 7484	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 7647	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp
PR 7800	core/libs/prx/libSceAgcDriver/Graphics/include/Context.hpp core/libs/prx/libSceAgcDriver/tests/BufferPool.cpp

[parl0v]

Windows data for this PR and #5987: RTX 4060 8 GB (NVIDIA 617.42), i5-14400F, 16 GB RAM, Windows 10 19045. Heap 0 reports a 7189 MiB budget, heap 1 (system memory) 7448 MiB. Minecraft Dungeons II (PPSA16064 01.100.100), first world load, main 5d9dcc4 + #6308 + #5372 + #6490, APS5_TEXTURE_CACHE_MIB=3072 (which oversubscribes this card; see below).

Two of three first loads died, both with VK_ERROR_OUT_OF_DEVICE_MEMORY on memory type 0, which on this driver is heap 1, not the device-local heap:

libs	result	last placement line before it	error
base	died in the loading screen	256 such allocations, 2218 MiB, heap 0 at 7620 of 7189 MiB	vkAllocateMemory device buffer: allocation=16777216 memoryType=0 reclaimed=0 initialResult=-2 attempts=1
+ #5987	died right after the intro cutscene	1024 such allocations, 4995 MiB, heap 0 at 8067 of 7189 MiB	vkAllocateMemory device buffer: allocation=134217728 memoryType=0 reclaimed=980363776 initialResult=-2 attempts=2
+ #5987 + this PR	reached the HUD and walked 320 s	512 allocations, 694 MiB, heap 1 at 5397 of 7448 MiB	none

So on Windows the refusal I saw was never of the device-local type; it was the host type, once DeviceMemoryType had placed the buffer there and heap 1 was full. With this PR's heap-1 figures in the placement line, heap 1 was already at 5155 of its 7448 MiB before the first placement: a lab counter on vkAllocateMemory put ~4.7 GB of that in host-pointer imports and ~1 GB in staging buffers, so a 16 GB machine has ~2.3 GB left for placements, and the second load placed more than twice that. #5987's fallback doesn't apply there, since it only retries a refused device-local type; this PR's "keep device-local when no host heap has room" is the case these loads hit.

I can't show this PR's branch catching it: in the third load and in a 3-round A/B from an in-world save (this PR on and off, #5987 alone vs #5987 + this PR) heap 1 peaked at 5286-5419 MiB, so no row logged kept in device-local memory and the arms ran the same (3/3 survived each, walk 4.95 FPS mean both). And with the cache budget left automatic (1797 MiB storage images here), the same scene places nothing at all, so on this card the pressure needs an oversized cache budget or a bigger scene.

AI-assisted: yes (Claude Code)

[fdfawef324]

Thank you for the Windows data, @parl0v. Two readings:

• Where the refusal came from. On your card the -2 was for memory type 0, which on your driver is heap 1, after heap 1 had filled: the host type DeviceMemoryType had placed the buffer in. fix(agc): place device memory in host memory when the driver refuses a device-local allocation #5987's fallback retries only a refused device-local type, so it does not cover that case; "keep device-local when no host heap has room" does. That is the failure mode this PR targets, seen on a different GPU and RAM size than ours (RTX 3060 12 GB, 32 GB), where our link to it is inferred from timing.
• What it does not show. In your A/B (this PR on and off, fix(agc): place device memory in host memory when the driver refuses a device-local allocation #5987 alone vs. fix(agc): place device memory in host memory when the driver refuses a device-local allocation #5987 + this PR, 3 of 3 survived in each arm) heap 1 peaked at 5286-5419 of 7448 MiB and no row logged kept in device-local memory, so the guard never ran. The one first load with the PR that survived is a single run. We count your runs as evidence of the failure mode, not yet as the guard preventing it, and added that to the description.

If you rerun anything: the first-load path is where #5987 alone died, so an on/off pair there (same APS5_TEXTURE_CACHE_MIB=3072) is the run that could show the guard. No obligation.

AI-assisted: yes

[blawesom]

Rebase handoff — rebased onto current main (all four commits, authorship unchanged). The conflicts were main's allocate() refactor meeting this series' inline creation: the merged version keeps main's structure and routes its allocation through your AllocateDeviceMemory fallback helper, so the host-memory fallback still applies; your buffer-pool reclaim and heap-room commits replay on top. One auto-merge had duplicated the mock device functions in DepthSurfaceReuse.cpp (removed).

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV):

• builds; tools/check_conventions.py --base origin/main passes
• full suite: 1338/1339 — the only failure is the known main regression (agc_driver_tessellation segfault, feat(shader): read hull inputs at a control-point index computed from data #4727), not this PR

To pick it up:

git fetch https://github.com/blawesom/AnyPS5 rebase/6553
git checkout <your branch>
git merge --ff-only FETCH_HEAD

Head is 321ab70c9.

[fdfawef324]

Thanks, @blawesom. I checked your offered 321ab70c9 rebase and retained its authorship/credit. Since then #5987 merged with a newer allocation path, so I replayed only this PR's host-room guard onto fc623324a and kept main's AllocateImageMemory routing, buffer-pool trim and texture-cache reclaim. The PR now has one guard commit; no open #5987 dependency remains.

The rebased tests cover the current helpers: when neither heap fits, a successful device allocation stays device-local; a mocked refusal still tries device-local before host memory for both buffers and images. A separate isolated red control, changing only the host-room lookup back to the old host lookup, fails all six budget/mask assertions. The unchanged source passes. The two fresh control executables plus both test runs took 2.81 seconds of bounded child-job time, without mutating the production worktree or its cache identity.

Your RX 9070 XT/RADV Linux 1338/1339 result remains attributed to your offered head. It is not a fresh result for this rebased head. The earlier Windows title evidence also stays qualified: @parl0v's A/B did not reach the guard, and this update does not establish a Demon's Souls memory or FPS improvement.

The normal Release targets built, and all five selected integrated CPU/mock CTests passed in 11.58 seconds with no skipped cases: memory fallback, buffer pool, its two small-device variants and graphics. Conventions pass; the exact source remains clean with TraceRevision 4e8edc80e264/dirty 0. Fresh split Vulkan adjacency targets are built and pinned.

Fresh current-head managed-slot Vulkan batch: agc_driver_pooled_buffer_range, agc_driver_sampled_cache_budget, agc_driver_sampled_cache_budget_memory and agc_driver_storage_cache_budget, 4/4 passed in 9.36 seconds, no skipped cases, with ANYPS5_REQUIRE_VULKAN=1. These are real Vulkan adjacency tests, not a real both-heaps-full reproduction. Actual executables and runtime libraries were hash-checked before and after. A clean merge into main 64b57d1b0 was checked separately; all tests ran on the exact candidate.

The updated PR is a draft while the shared-area landing order is coordinated; the guard itself has no build dependency on those branches.

AI-assisted: yes.

Co-Authored-By: Codex noreply@openai.com