fix(libc): give the scripting heap hooks blocks from libc's own heap

[milescoviello]

What

Since #5092 (merged 10-15), Metal Gear Solid Master Collection (PPSA18194) dies 0 s in on main: EXIT-139 with no message, 5 of 5 runs on 82ddae7 (one of them with #6814 applied). Before #5092 its Unity launcher ran. In the 10-15 sweep (0029e62) it reached its LoadExec into mgs4/mgs4.elf at 30 s, a 30 s run on that build reaches the limit, and with #5092 reverted on 82ddae7 it reaches the limit 2 of 2.

What happens:

1. The title ships its own libc.prx. At start, that libc initializes the title's malloc replacement, and Unity puts its heap in a 256 MiB direct mapping at 0x200000000.
2. IL2CPP's GC looks up scriptingGetMem by name (the only hook name in Il2cppUserAssemblies.prx) and now finds feat(libc): provide the runtime scripting allocator hooks #5092's hook. The hook allocates through ApplicationHeapAlign. That heap is not initialized in this process (_init_env belongs to the title's own libc), so the lazy initialization from feat(libc): lazily initialize application heap on first access #4774 reads the same replacement table from the process parameters and calls its initialize callback a second time.
3. Unity starts a second heap. With APS5_TRACE_MEMORY=1, a new 256 MiB mapping at 0x215000000 appears right before the four scriptingGetMem(0x1000, 0x40000) calls. The run with feat(libc): provide the runtime scripting allocator hooks #5092 reverted makes no such mapping.
4. The next free of a block from the first heap fails dlmalloc's least_addr check (the core has mspace_free(0x215000010, 0x200001060)). The player then calls the title libc's abort (int 0x45), which ends the process with SIGSEGV.

How

The two hooks now allocate and free through libc's own guest heap (GuestHeapAlign / GuestHeapFree) instead of the title's registered allocator, so calling them never initializes that allocator. The alignment rules stay as #5092 made them. A size that overflows now returns null instead of throwing. The debt entry records why, and that which console library exports the hooks is still unknown.

Tested

Ubuntu 24.04, GCC 13.3, Release, RTX 4070 Ti (NVIDIA 595.99).

• New application_heap_scripting: with the process parameters installed and the heap not yet initialized, two scriptingGetMem(0x1000, 0x40000) blocks are 4 KiB-aligned, distinct and writable. Neither they nor their frees reach the registered allocator, and _init_env afterwards initializes it exactly once.
• The replacement-mode assertions now check that the hooks do not reach the registered allocator: it sees no size and no free, and failing it does not affect them.
• With main's ApplicationHeap.cpp, application_heap_scripting and application_heap_replacement fail. With the change, all 9 application_heap_* tests pass 3 of 3, and the 396 tests matching application_heap|guest_|libc|getifaddrs|heap pass.
• MGS4 Master Collection, pad input. On 82ddae7 it is EXIT-139 at 0 s. With this PR, 3 of 3 30 s runs reach the limit with the same log as the 10-15 build (542–553 lines, the same textures and asset bundles), and a 120 s run stops where the 10-15 sweep did: at 30 s, on sceWebBrowserDialogOpenForPredeterminedContent not implemented (open feat(libSceWebBrowserDialog): implement sceWebBrowserDialogOpenForPredeterminedContent #5904 and feat(libSceWebBrowserDialog): finish PredeterminedContent without UI (PPSA34015) #6092).
• Throwaway builds while narrowing it down:

scriptingGetMem variant	MGS4
#5092 (title allocator), blocks zero-filled (they already were)	EXIT-139 0 s
title allocator, 64 KiB padding after each block	EXIT-139 0 s
title allocator, 64 KiB padding before each block	EXIT-139 0 s
fresh anonymous mappings	30 s limit, 21 calls, no scriptingFreeMem

So the contents and bounds of the blocks don't matter; reaching the title's allocator does.

@mrfelek found the same crash in Hollow Knight: Silksong (PPSA12544), another IL2CPP title with its own libc.prx. On main af7db81 it segfaults about 30 s in. With #5092 reverted, or with this PR (35ea8317e), it runs at about 1,050–1,200 presents per 10 s. Their git bisect --first-parent also lands on #5092's merge.

@karimibrahim07 checked it on Windows (Windows 11, WinLibs GCC 15.2.0, Release). All 9 application_heap_* tests pass 3 of 3. With main's ApplicationHeap.cpp, application_heap_scripting and application_heap_replacement fail, the same two as on Linux.

@utajum confirmed it on God of War: Sons of Sparta (PPSA28997) on Linux with RADV (Ryzen Z2 Go, Mesa 26.2.0-devel), on main e09dbc2. That title ships its own libc.prx, so it is this PR's case, not #694's. On main it segfaults at start-up right after global-metadata.dat opens, in the same il2cpp_init → title free → title abort chain as MGS4. With this PR's ApplicationHeap.cpp it boots, loads a save and plays, in 10 runs of 75–180 s with no crash.

@imellag checked it on macOS (Apple M4 Pro, Rosetta 2, MoltenVK 1.4.2), with this head merged into #7332, which adds the macOS runtime. The 9 application_heap_* tests pass, and with the earlier ApplicationHeap.cpp the same two abort. God of War: Sons of Sparta segfaults about a second after global-metadata.dat opens without this PR, in the same chain, and with it the game reaches gameplay.

Not checked: a Unity title that runs on AnyPS5's libc, whose GC now gets its blocks from libc's heap instead of the title's allocator.

#6713 edits the same files for executables without process parameters; it does not touch these functions. @karimibrahim07 moved #6713's test block and debt line (668d51a), so the two merge cleanly in either order.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA18194 (Metal Gear Solid Master Collection launcher)
• GPUs tested: NVIDIA GeForce RTX 4070 Ti, driver 595.99, Ubuntu 24.04
• AI-assisted: yes

🤖 Generated with Claude Code

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 3987	docs/dev/debt/libc.md
PR 5021	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 5303	docs/dev/debt/libc.md
PR 6044	core/libs/prx/libc/CMakeLists.txt core/libs/tests/ApplicationHeap.cpp
PR 6079	core/libs/prx/libc/CMakeLists.txt
PR 6226	docs/dev/debt/libc.md
PR 6469	docs/dev/debt/libc.md
PR 6975	docs/dev/debt/libc.md
PR 7039	docs/dev/debt/libc.md
PR 7081	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 7084	core/libs/prx/libc/CMakeLists.txt
PR 7201	core/libs/prx/libc/CMakeLists.txt core/libs/tests/ApplicationHeap.cpp
PR 7386	docs/dev/debt/libc.md
PR 7417	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 7473	docs/dev/debt/libc.md
PR 7525	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 7715	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md

[milescoviello]

An independent bisect agrees. I bisected main (first-parent, 115 merges between b4a4fca and 1b1f848) for this title's start-up crash, classifying each run by whether the fault is in the title's sce_module/libc.prx.guest.prx. It lands on #5092 (7663bc6) as the first bad merge; c10aa03, the merge before it, runs.

On current main bc3c0fc with #6814 and #6237 merged (Linux, RTX 4070 Ti, pad input), the title still dies at 0 s with the fault in its own libc. With this PR added (35ea831, on the slightly newer main 0dc1562 with the same two PRs), 2 of 2 runs reach the 180 s limit, and the launcher's LoadExec into mgs4/mgs4.elf happens in both. #5904 is on main now, so this PR and #6237 are what stand between main and the game here.

AI-assisted: yes

[mrfelek]

Hollow Knight: Silksong (PPSA12544), another IL2CPP title that ships its own libc.prx, has the same startup crash since #5092, and this PR fixes it.

Linux, Ryzen 9 5950X, RTX 3080 Ti, NVIDIA 615.71.09, GCC 15.3, Release:

Build	Result
main af7db81e6	segfault about 30 s in, right after global-metadata.dat is opened, no frames
main af7db81e6 with aea1dd601 (#5092) reverted	runs, ~1,050–1,200 presents / 10 s
main af7db81e6 + this PR (35ea8317e)	runs, ~1,080–1,200 presents / 10 s

A git bisect --first-parent from 14036cb0e (Oct 13, good) to af7db81e6, keeping the Oct 13 relink fixed, also lands on the #5092 merge (7663bc63b).

AI-assisted: this comment was drafted with the help of Claude Code and checked against the logs by us.

[milescoviello]

Thanks @mrfelek. A second IL2CPP title, with an independent bisect that lands on #5092, makes this much clearer. I added it to the description.

AI-assisted: yes (Claude Code)

[karimibrahim07]

Windows check, since the description lists it as not checked. Head 35ea8317e merged onto main e3a8e4430 (clean merge). Windows 11 Pro (26300), Ryzen 5 3600, WinLibs GCC 15.2.0 r7, Ninja Release.

• All 9 application_heap_* tests pass, --repeat until-fail:3, application_heap_scripting included.
• Negative control: with main's ApplicationHeap.cpp and this PR's test, application_heap_scripting and application_heap_replacement fail (0xc0000409), the same two tests as on Linux.

Overlap with #6713 (mine): the bot found a textual conflict in ApplicationHeap.cpp (tests) and docs/dev/debt/libc.md, because both PRs added a block right after the lazy case and a line under the scriptingGetMem entry. I moved #6713's test block and debt line away from yours (668d51a), so the two now merge cleanly in either order. With both merged onto main, all 10 application_heap_* tests pass 3 of 3. As you say, #6713 doesn't touch these two functions: it only changes ApplicationHeapInitialize for a null process-parameters pointer. With both applied, the hooks still never reach the registered allocator.

Code: routing the hooks to GuestHeap instead of ApplicationHeapAlign makes sense given the MGS4 and Silksong results: those titles' own libc owns the registered allocator, and the hooks were initializing it a second time. Catching std::length_error alongside bad_alloc matches the new SIZE_MAX check. I have no code requests.

Not checked here: a title. I don't have MGS4, Silksong or a Unity title that uses AnyPS5's own libc (the PPSA28997 case you mention). For that case the GC's blocks now come from libc's heap instead of the title's allocator, so a run of it would be the useful remaining evidence.

AI-assisted: yes. I ran the build, the tests and the negative control above.

[milescoviello]

Thanks @karimibrahim07 for the Windows run and the negative control, and for moving #6713's block so the two merge in either order. I added it to the description.

AI-assisted: yes (Claude Code)

[utajum]

Confirmed on PPSA28997 (God of War: Sons of Sparta), Linux, RADV (Ryzen Z2 Go / REMBRANDT, Mesa 26.2.0-devel), main e09dbc2.

PPSA28997 ships its own sce_module/libc.prx, so it is the title-libc case, not the #694 one.

• main: SIGSEGV at start-up, right after global-metadata.dat opens. Core: il2cpp_init -> title libc free (tIhsqj0qsFE) -> eboot allocator -> title libc abort (L1SBTkC+Cvw). Same chain as MGS4.
• main + this PR's ApplicationHeap.cpp: boots, reaches the title screen, loads an existing save and plays. 10 runs of 75-180 s, no crash.

[milescoviello]

Thanks @utajum, that's a useful run. It's a third IL2CPP title with its own libc.prx, after MGS4 and Silksong, and it's also the first RADV result for this PR. I'd listed PPSA28997 as the #694 case under "Not checked", which was wrong. I've corrected that and added your run to the description.

AI-assisted: yes (Claude Code)

[imellag]

Checked on macOS: Apple M4 Pro, macOS 27.0.1 (26A434), Rosetta 2, Apple clang 21.0.0, Vulkan SDK 1.4.363.0 (MoltenVK 1.4.2). main has no macOS runtime yet, so this is head b4e129ff8 merged into #7332 (690c044a6 merged into main a62c724c3), no conflicts, x86-64 build under Rosetta.

• application_heap_*: 9/9, and again with --repeat until-fail:3. With the merge's ApplicationHeap.cpp swapped for the one before this PR, application_heap_replacement and application_heap_scripting abort, the same two as on Linux and Windows.
• God of War: Sons of Sparta (PPSA28997, 01.008.001): without this PR the process ends with SIGSEGV about a second after global-metadata.dat opens, with nothing in the log. A temporary report in the write watch's fault handler (local only) shows the chain described above: the fault is the int $0x45 in the title libc's abort (L1SBTkC+Cvw), called from the eboot under the title libc's free (tIhsqj0qsFE), called from Il2cppUserAssemblies. With this PR the title starts, loads its FMOD banks and reaches gameplay. That run also needs a local workaround for a separate macOS module-loading stop, reported on feat: macOS on Apple Silicon host support (Rosetta 2 and MoltenVK) #7332.

Not checked: other titles on macOS.

AI-assisted: yes (Claude Code). I ran all of the above on my machine.

[milescoviello]

Thanks @imellag for the macOS run and the Sons of Sparta check. I added it to the description.

AI-assisted: yes (Claude Code)

[adriadam10]

Checked the two hooks against GuestHeap (alignment and overflow paths: length_error is thrown at GuestHeap.cpp:75/266 and now returns null), the new application_heap_scripting test (registered in ctest and in the Windows PATH list; it pins no allocator init/free before _init_env, so it fails on main), and the updated libc debt entry. Windows, macOS and three titles confirmed by the reviewers above; merged.

[madkyp]

Also fixes Plants vs. Zombies™: Replanted (PPSA29802, #6404): on Linux 47a266c it aborted in the title's free() (chunk from the first application heap at 0x200000000 freed into the second at 0x215e00000). On main 7a015c0 it reaches the profile menu and runs 3 min without a fatal. Ryzen 7 9800X3D + RX 9070 XT (RADV, Mesa 26.2.4), CachyOS.

[milescoviello]

@madkyp thanks for the report. Good to know Plants vs. Zombies: Replanted (PPSA29802) had the same cross-heap free() abort (a chunk from the first application heap freed into the second) and now reaches the profile menu on main 7a015c0 and runs 3 minutes without a fatal. #6404 returns a 404 for me, so if it lives somewhere else (a discussion?), feel free to link this there.

AI-assisted: yes (Claude Code)