fix(libkernel): replace anonymous Windows stack mappings

[dogukanatlihan]

What

Fix the DestroyJavaVM stack-guard cleanup warnings in Java 25 LAN / DNS, PPSA99869 (#5501), reported in #5918. HotSpot requests mmap(addr, size, PROT_NONE, MAP_PRIVATE | MAP_ANON | MAP_FIXED, -1, 0) to uncommit the guard range. The current path rejects native Windows stacks outside the guest arena.

Permit fixed anonymous POSIX mappings within one current or registered native Windows stack. Decommit and recommit the pages while retaining the reservation, so a later readable mapping sees fresh zero-filled memory. Check ownership, allocation boundaries and native PAGE_GUARD pages before mutation; reject current-stack ranges ending within 64 KiB of the active host frame. Other mapping APIs keep their existing behavior. Preserve protection intervals on either side of a replacement and recommit tracked pages during thread cleanup.

The pinned OpenJDK BSD implementation and the homebrew's reservation-preservation patch establish the request; disassembly of the public release confirms flags 0x1012. Changing protection alone would retain old contents.

This is separate from #5918's stack-bound reporting and #6443's mprotect change. The repository overlap checker reports two textual conflicts: #6091 adds <memory> where this patch adds <functional> in Pthread.hpp (keep both), and #6469 edits the same fixed-mmap debt paragraph for its independent low-mapping-hint change (preserve both descriptions). Neither is a production dependency. #5703 is already merged. #6452 is a Windows full-build test-fixture prerequisite, not part of this production patch.

Tested

Windows 11 Pro 26300, i7-14700K, WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON.

• Fresh full Release build passes on main 44df21632125dbe3113a19173e7ebb879da53003 plus this commit (57c1ca733) and the existing test(agc): align the indirect arguments block at run time #6452 AGC test-fixture patch (4fe92c9c2). All 34 selected memory/thread tests pass, no skips, in 6.56 seconds. The seven built source/documentation files match this commit after line-ending normalization; test(agc): align the indirect arguments block at run time #6452 is the only additional change in the test tree. Conventions and whitespace checks pass.
• Earlier negative controls: unchanged libkernel fails the first valid stack-replacement assertion; restoring the old protection-interval erasure makes the counted pthread-key destructor fail before Windows releases the native stack.
• The regression covers exact addresses, full-range zero-fill, adjacent canaries, PROT_NONE, repeated RW replacement, invalid requests, native guard preservation, active current frames, synchronized peer stacks, and partial-range cleanup.
• Public LAN_DNS_Probe-v01.000.000: initialize, create VM, destroy VM. Baseline records two Attempt to deallocate stack guard pages failed warnings; patched records none. Both return 0 from JNI_CreateJavaVM and complete DestroyJavaVM. Only libkernel.prx differs across 174 runtime files. These before/after runs used fa9365ca7 plus test(agc): align the indirect arguments block at run time #6452; all affected source base blobs match current main 44df21632 and the tested patched sources match this patch. Full reproduction and results.
• Linux GCC syntax checks passed for the changed translation units and regression test. Linux runtime, PS5 hardware, failed-recommit injection, concurrent retirement stress, and natural Windows stack growth were not tested. The guard regression uses an explicitly guarded stack page. Peer-stack replacement requires caller coordination.
• LAN/DNS operations were not exercised. The harness terminates the still-open app after recording VM destruction; no clean application-exit or performance claim is made.

Targeted test command:

ctest --test-dir build --output-on-failure -R "^guest_.*(memory|mmap|mprotect|munmap|stack|pthread|thread_lifecycle|thread_exit)"
100% tests passed, 0 tests failed out of 34

Checklist

• Based on current main; no other open PR implements this stack-replacement behavior (related overlaps listed above)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw; no silent stubs added
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source (none added)
• Generic behaviour, not specific to one title
• Depends on: test(agc): align the indirect arguments block at run time #6452 (Windows full-build test fixture only)
• Unblocks: PPSA99869 Java 25 LAN / DNS (PPSA99869) #5501 (fixes the observed VM-cleanup warnings)
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6469	docs/dev/debt/libkernel.md L67

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 3986	core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 4449	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
PR 5021	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 6044	core/libs/tests/GuestKernelStackProtect.cpp
PR 6149	docs/dev/debt/libkernel.md
PR 6446	docs/dev/debt/libkernel.md
PR 6519	docs/dev/debt/libkernel.md
PR 6879	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 6908	docs/dev/debt/libkernel.md
PR 6936	docs/dev/debt/libkernel.md
PR 6977	docs/dev/debt/libkernel.md
PR 7039	docs/dev/debt/libkernel.md
PR 7049	docs/dev/debt/libkernel.md
PR 7075	docs/dev/debt/libkernel.md
PR 7088	docs/dev/debt/libkernel.md
PR 7105	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/DirectMemory/DirectMemory.hpp docs/dev/debt/libkernel.md
PR 7121	docs/dev/debt/libkernel.md
PR 7129	docs/dev/debt/libkernel.md
PR 7149	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/DirectMemory/DirectMemory.hpp core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 7192	docs/dev/debt/libkernel.md
PR 7201	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/Pthread/src/Thread.cpp core/libs/tests/GuestKernelStackProtect.cpp
PR 7332	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/DirectMemory/PosixMemory.cpp core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp core/libs/tests/GuestKernelStackProtect.cpp docs/dev/debt/libkernel.md
PR 7509	docs/dev/debt/libkernel.md
PR 7565	core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md

[jotitadev]

Linux run (Ubuntu 26.04, Intel i7-8700, GCC 15.2, full build, non-GPU tests), head 57c1ca733 merged onto main ce991f1ec. The description says Linux runtime was not tested.

• Builds; the 35 tests matched by the description's ctest -R filter pass. That includes guest_kernel_stack_protect, whose non-Windows part is now CheckGuardCleanup(GuardMode::Protect): the pthread-key destructor finds the guard pages writable again. It passes 30 of 30 repeated runs.
• With the five libkernel files from main (the PR's test kept), the same 35 tests also pass. That is expected: the new behaviour is inside _WIN32, and the Linux part of the test only moved into the key destructor.
• Merged together with #6542, which also changes mmap_nid_postfix and DirectMemory.cpp (the overlap bot does not list this pair): the merge is clean, it builds, the same filter plus the shm tests (42) passes, and the non-GPU suite gives the same result as main. This covers only the Linux side of the combination.

Non-GPU suite (ctest -E "^agc"): same result as main; only file_writer_binary and file_writer_text fail, and they fail on main too on this machine (Ubuntu 26.04 coreutils).

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[karimibrahim07]

Windows check of head 57c1ca733, merged onto main 5d9dcc42d (clean merge; it also merges cleanly with #6443). Windows 11 Pro (26300), Ryzen 5 3600, RTX 3060, WinLibs GCC 15.2.0 r7, Ninja Release.

Title: Java 25 JVM Startup (PPSA99865, 01.000.023), relinked with --windows --to-intel --trap-sdk-syscall-stubs --registry, all four rows:

• On main, DestroyJavaVM prints Attempt to deallocate stack guard pages failed three times (the main thread at 0x410000 here, and two VM threads).
• With this PR it prints none. JNI_CreateJavaVM returned 0, Hello25 returned normally and DestroyJavaVM complete as before.

Tests: guest_kernel_stack_protect, guest_thread_self and the other memory tests I selected pass, 5 of 5.

Code:

• I checked the lock order. ReplaceGuestThreadStack holds stackLock while the callback takes g_protectionLock. Every other holder of g_protectionLock (RecordProtection, GuestProtection, GuestPageTableUsage) is a leaf, and DoMprotect takes stackLock (in NoteGuestStackProtection) and g_protectionLock one after the other, not nested. So I see no inversion.
• One question about the peer-stack case. When the range is on another live thread's stack, the callback MEM_DECOMMITs and recommits it while that thread may be running, and the description says peer-stack callers "must coordinate with the peer". HotSpot only does this on a thread's own stack at teardown, so this title doesn't exercise it. Would it be safer to refuse a peer's stack (EINVAL) unless the peer is known to be stopped, rather than rely on the caller?

AI-assisted: yes. I ran the build, the tests and the title above.