Windows: guest_kernel_stack_protect aborts since #3788, sceKernelMprotect accepts a range crossing a guest stack's lower bound #4088

Environment: Windows 11 Pro 10.0.26220, WinLibs GCC 15.2.0 r7, Ninja, Release, BUILD_TESTING=ON; main 1a7b96b5b plus #4021 (needed for libkernel to compile).

Reproduction

cmake --build build --parallel
ctest --test-dir build -R '^guest_kernel_stack_protect$' --output-on-failure

First bad merge: dc70a66cb, #3788 "fix(libkernel): allow sceKernelMprotect on committed host memory outside the guest registry" by @xXPBSXx. Its parent 3929c21fb passes; the merge fails.

Failure

CTest reports Exit code 0xc0000409. The test prints:

FATAL: abort() (SIGABRT) on thread 50420
    #0 0x00007ff86b30af47 libc.prx+0xaf47
    #1 0x00007ff86b3a04d9 libc.prx+0xa04d9

The test's Require calls a bare std::abort() with no message. I rebuilt the test source with Require printing its line, plus a VirtualQuery of the range. Output:

start=000000fc00100000 end=000000fc00200000 outside-Page=000000fc000fc000 alloc=000000fc00000000 base=000000fc000fc000 region=0x104000 state=0x1000 type=0x20000 prot=0x4
Require failed at line 84: mprotect_nid_postfix(reinterpret_cast<void*>(outside - Page), 2 * Page, 3) == -1

The abort site is the last check in Worker, core/libs/tests/GuestKernelStackProtect.cpp:84. A protection change on [start - 16 KiB, start + 16 KiB), which crosses the stack's lower bound, must fail. It now returns 0.

Analysis

On Windows, a guest thread runs on its host thread's stack. _beginthreadex is given guest stackSize + HOST_STACK_MARGIN (1 MiB), and the stack is committed as one MEM_PRIVATE / PAGE_READWRITE region. The guest stack is the upper part of that region, and the 1 MiB below it is the host margin. Above, the stack is 0xfc00100000..0xfc00200000; the committed region starts at allocation base 0xfc00000000.

#2982 (@TheAggie) let DoMprotect change pages inside a live guest stack directly. Its commit message states: "Ranges that cross the stack's bounds are still rejected." The test pins that rule.

#3788 added a Windows fast path in DoMprotect (core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp:924):

if (memory.Type == MEM_PRIVATE && memory.State == MEM_COMMIT && memory.RegionSize >= bytes && !mutation.Overlaps(pointer, bytes)) {
    if (mprotect(const_cast<void*>(pointer), bytes, nativeProtection) != 0) throw ...;
    return 0;
}

The crossing range skips the stack branch because it is not inside [stackStart, stackEnd). It is not a registered guest allocation either. Before #3788 it fell into mutation.Protect, which threw "not registered", so mprotect_nid_postfix returned -1. Now VirtualQuery at start - 16 KiB sees a committed private region of 0x104000 bytes, which is at least the requested size, so the fast path calls VirtualProtect and returns 0.

This only affects Windows: the fast path is inside #ifdef _WIN32.

The test failure points to a wider problem. The fast path lets a guest change the protection of any committed private host memory that is not in the guest registry. That includes the 1 MiB host-stack margin below every guest stack (PROT_NONE there would crash host code that runs deeper on that thread), other threads' stacks, and host heap pages. It also returns before RecordProtection and NoteGuestStackProtection, so those changes are not recorded and are not restored when the thread exits.

The motivating case in #3143 (RE4 PPSA07411, 0xf8380000+0x580000, prot 0xc2) was described as a thread stack. That range skipped #2982's stack branch only because its protection had the GPU bits set ((prot & GuestProtGpuReadWrite) == 0 is false).

Possible minimal fixes (not built or tested)

1. Narrowest: keep the fast path, but skip it whenever the range touches a live guest thread stack or its host region. For example, skip if GuestThreadStack(first, ...) or GuestThreadStack(end - 1, ...) is true. The test's crossing range would then hit GuestThreadStack(end - 1) and fall back to the rejecting path. A range lying entirely inside the host margin would still pass, so a stronger check compares memory.AllocationBase with the allocation base of each liveStacks entry.
2. Closer to what libkernel: AvailableDirectMemorySize reports first gap, Mprotect rejects host stack memory #3143 needed: remove the general host-memory fast path. Let fix(libkernel): change the protection of guest thread stack pages #2982's in-stack branch also accept ranges that request GPU access (apply the CPU part natively and record the protection). Ranges outside a guest stack would then keep rejecting unregistered host memory.

Option 2 needs confirmation that the RE4 range lies inside a single guest stack. That is not settled by #3143.

Found and bisected with Claude Code assistance (five full Windows builds, each with #4021 applied to make libkernel compile); reviewed before posting.