fix(libkernel): reject malformed ranges in sceKernelReleaseDirectMemory

[Pouare514]

What

sceKernelReleaseDirectMemory only rejected a negative start and a zero length, so malformed ranges reached the physical pool: a sub-page length silently freed a whole page while fragmenting the block map at byte granularity, and an out-of-range length walked the page table past the backing store. Validate page alignment of start and length plus the range bound up front and return EINVAL instead, mirroring DoMunmap.

Tested

• New CheckReleaseDirectMemoryRejectsInvalidRanges in core/libs/tests/GuestMemory.cpp: negative start, zero/misaligned/huge/out-of-range lengths.
• guest_memory passes with the fix; with the fix stashed it aborts (0xc0000409), so the test pins the behavior.
• CI green: conventions, eligibility, intake, report, relinker macos/ubuntu/windows (linux/windows jobs skip by design).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

[popcoder565]

Native Windows run, since no Windows results were posted. Setup: Windows 10 (19045), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release. Head aa77b2b8, merged onto main 8b393b6 in one build with 31 other PRs that had no Windows run.

• This PR's tests pass: guest_memory.
• Full ctest in that build: 520 of 521. The one failure is guest_offline_imports, broken by feat(libSceRudp): implement sceRudpGetStatus #2359 (reported there).

No title was run.

🤖 Addressed by Claude Code

[Kikoarl]

Tested on Linux (CachyOS x86_64, Intel RPL-S iGPU / Mesa 26.3.0-devel / Vulkan 1.4.363, GCC 16.2.1) at aa77b2b8:

• Build: Built cleanly in RAM (/tmp/anyps5_build) with zero warnings/errors.
• Conventions: python3 tools/check_conventions.py --base origin/main --head HEAD -> PASS (0 errors).
• Verification: Verified GuestMemory.cpp (CheckReleaseDirectMemoryRejectsInvalidRanges). Page-alignment checks and bounds validation against DIRECT_MEMORY_SIZE cleanly reject unaligned and out-of-bounds ranges with SCE_KERNEL_ERROR_EINVAL.

LGTM!

[oneandonlydean]

Conflicts with main in core/libs/prx/libkernel/DirectMemory/Export.cpp after #2236; please rebase.

[adriadam10]

Rebased onto current main (conflict with #2236 in sceKernelReleaseDirectMemory: the range validation now runs before DirectMemoryViews). Build and ctest green on the rebase, including guest_memory. Page alignment matches KytyPS5 and shadPS4 (EINVAL); merging.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#1905	core/libs/tests/GuestMemory.cpp L7
#1961	core/libs/tests/GuestMemory.cpp L7-8

[adriadam10]

The only red check was write-watch (macos-15-intel): write_watch_threads timed out at 30 s on a slow runner (the same job on macos-15 passes, and those tests don't touch libkernel). Re-running it; merging once green.