fix(libkernel): place sceKernelReserveVirtualRange at its address hint

[TheAggie]

What

sceKernelReserveVirtualRange dropped its address unless MAP_FIXED was set. 1d806a4 made fixed reservations land at their address and left the others "placed as before", which means at the lowest free address. Without MAP_FIXED, the address is a hint, as in FreeBSD's mmap: the reservation goes at the hint when that range is free, and otherwise at the first free range above it. This passes the hint to MapAligned, which already places a non-fixed mapping that way. Fixed reservations are unchanged.

Found with RetroArch (PPSA99169). Its PS5 platform layer reserves the cores' direct-memory heap at a hint near 0x2000000000, outside the GPU window (0x200000000-0x300000000), and refuses a range inside that window. With the hint dropped, every reservation landed at 0x200400000, the heap was never created, and the Dolphin core's first aligned allocation returned null.

Tested

Windows 11 Pro (26200), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release.

• New check in guest_memory, CheckHintedVirtualReservation: a reservation at a free hint lands exactly there, and one whose hint is taken lands above it, not below. With the old line it fails (GuestMemory.cpp:477, the reservation that should land at its hint).
• On this machine guest_memory already stops later, at CheckGuestModuleImageProtection (its dlopen of the test module returns null). That happens on clean main cf8b129 too; the new check runs before it and passes.
• Rebased onto main cf8b129. Full suite: 550 of 554 pass, 1 skipped. Besides guest_memory, agc_driver_prepared_shaders_debug_failure also fails on clean main on this machine, and guest_module_identity failed once and passed when rerun alone. It was run from the generated CTestTestfile.cmake commands because antivirus blocks ctest.exe on this machine.
• tools/check_conventions.py passes.
• RetroArch's Dolphin core got its heap and booted the GameCube 240p Test Suite with this change, on a local branch that also carries other unmerged changes the core needs. RetroArch doesn't reach the core on main alone.

Not tested on Linux or macOS.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none. fix(libkernel): map direct memory as write-watched private pages until it is aliased #2054 also edits DirectMemory.cpp and GuestMemory.cpp but not the reservation code. It conflicts with current main in those files by itself.
• AI-assisted: yes

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#1961	core/libs/tests/GuestMemory.cpp L468, L1528

[fma-rc]

Checked merged onto main 78aedb4 (Linux, GCC 16.2.1, Release):

• guest_memory passes. With only DirectMemory.cpp reverted it fails in 3 of 3 runs at GuestMemory.cpp:477 (Require(placed == hint)), so the new test does cover the change.
• Full ctest on an NVIDIA RTX 5080 (615.78.08) and on RADV (Radeon 610M, Mesa 26.1.4): no failure that main doesn't already have.