fix(libkernel): reprotect bundled guest module images on Windows

[tv7]

What

Fixes item 4 of #1794. On Windows, sceKernelMprotect accepted only the executable's image and threw Memory protection of a foreign image is not supported for any other image. Bundled title modules are relinked into DLLs named <module>.guest.prx, so a title that reprotects pages of its own bundled module could not start. Dying Light 2's bundled libNRD does this at start-up (prot=0x12 over 0x11278 bytes, then 0x11 over 0x178540 bytes).

DoMprotect now also accepts images whose file name ends in .guest.prx. It registers their committed pages in GuestAllocations the same way as the executable's: RegisterMainImage's Windows walk becomes a per-image registration, and the executable is one caller. Any other image (system DLLs, AnyPS5's own .prx libraries) is still refused.

Tested

Windows 11, MinGW GCC 15.2.0, RTX 3070:

• guest_memory has a new Windows case. It loads a small DLL named guest_memory_module.guest.prx and reprotects a data page read-only, then writable (writes and checks a byte), then read-only again, checking the host protection each time. It also checks that a page of kernel32.dll is still refused. Without the fix it fails with Memory protection of a foreign image is not supported; with it, it passes.
• Full build and ctest: 477/477 pass.
• python tools/check_conventions.py --base origin/main passes.
• Dying Light 2 (PPSA02262): both libNRD reprotections now succeed, and the title continues to its own Net_StartUp failed assertion (item 5 of Dying Light 2 (PPSA02262) on Windows: what stops it between conversion and its own startup #1794, unrelated to this change). The other items of Dying Light 2 (PPSA02262) on Windows: what stops it between conversion and its own startup #1794 were worked around locally for this run and are not part of this PR.

Linux is unchanged. I did not build or run it.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

🤖 Generated with Claude Code

[MarcZX10]

I found a module-lifetime issue in the new registration. registerImage adds committed .guest.prx pages to the global allocation registry as non-releasable and records the base in registeredImages, but guest modules can later be unloaded: sceKernelStopUnloadModule calls dlclose_nid_postfix, whose Windows Module destructor calls FreeLibrary. Nothing removes those registry ranges or the base from registeredImages on unload. This leaves dead image ranges registered, can make later mappings at the released address fail, and can expose stale addresses to GPU builds. Please add unload cleanup coordinated with outstanding allocation leases and cover mprotect → unload → reload/address reuse in a Windows test before approval.

[tv7]

Thanks, you're right: nothing undid the registration when a module was unloaded. Fixed in the new commit:

• GuestAllocations records each registered image's extent and gets GuestAllocationsUnregisterImage_nid_postfix. It waits for outstanding leases on that extent through RequireUnpinned (same path, pin waiter and timeout as unmap/protect), reports the extent to the invalidator, then drops the image's non-releasable ranges and its registeredImages entry.
• The Windows Module destructor in DynamicLoader.cpp unregisters the image before FreeLibrary. If the wait fails (lease still held after APS5_PIN_WAIT_MS), the module stays loaded, the destructor prints the error, and it returns, so GPU work never sees a freed image.
• If FreeLibrary only drops a reference and the image stays mapped, the next sceKernelMprotect on it registers it again from the current page protections.

The test (guest_memory, Windows) now loads the fixture through dlopen_nid_postfix and checks:
mprotect → registered → dlclose while a lease is held (it blocks, image still mapped) → lease dropped → dlclose returns, range no longer registered → a new allocation at the freed address registers → reload → mprotect registers again → dlclose → unregistered.

With the unregister call removed, the test fails at the "dlclose blocks while leased" check.

Full build + ctest: 477/477 pass (Windows, MinGW, RTX 3070).

[oneandonlydean]

~Module in DynamicLoader.cpp catches the UnregisterImage failure, prints it and leaves the module loaded. Failures need to stay explicit here; please unregister in dlclose_nid_postfix before the Module is destroyed and let the exception propagate.

[tv7]

@oneandonlydean done in 0040990:

• ~Module no longer touches the registry; it only frees the library.
• dlclose_nid_postfix calls UnregisterImage before erasing the handle, so a failure (image still leased) propagates and the handle, symbols and protection stay valid. The handle is erased and the library freed only after unregistering succeeds.
• If another open handle refers to the same image (LoadLibrary refcount), the image stays registered until the last handle closes.
• guest_memory: dlclose under a held lease now has to throw and leave the module usable and protected (APS5_PIN_WAIT_MS=1000 in that test), then the existing close-after-release, reload and address-reuse checks run.

Full build + ctest 477/477 on Windows (GCC 15.2 UCRT, RTX 3070); check_conventions.py passes.

[oneandonlydean]

Conflicts with main in core/libs/tests/GuestMemory.cpp; please rebase. The dlclose change is right.

[adriadam10]

Rebased onto current main as requested (single conflict in core/libs/tests/GuestMemory.cpp, main(): kept the _putenv_s under _WIN32 and the two new checks from main). Built and ran guest_memory and the libc/libkernel tests on Linux: green. The Windows case needs the Windows job; merging once CI is green.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#861	core/libs/tests/GuestMemory.cpp L536, L1269-1270
#1905	core/libs/prx/libc/include/GuestAllocations.hpp L60-61
#1961	core/libs/prx/libc/include/GuestAllocations.hpp L60-61
#2254	core/libs/tests/GuestMemory.cpp L21, L1269