fix(linux): enter the guest entry on a main stack at the console's address

[hugodearaujo]

What

On Linux the relinked executable entered the title's entry point on the host process stack, near the top of the address space (0x7ffc...). On the console a process's main thread stack sits inside the user area: a payload on a PS5 (FW 13.60) reports its main thread stack at 0x7eedfc000-0x7eeffc000 through pthread_attr_get_np, right above a worker thread's stack at 0x7eede8000-0x7eedf8000.

When the executable loads libkernel.prx, its Linux entry stub now calls EnterGuestMainStack_nid_no_patch (libkernel) through a GLOB_DAT slot, with the initial stack pointer and the entry point.

• The helper maps the main stack in the guest arena so that it ends where the console's does (0x7eeffc000, moved by the arena's offset in ASan builds). Below it is a 16 KiB guard. Its size is the host's RLIMIT_STACK (8 MiB when unlimited, at most 1 GiB).
• It calls the entry point there with the same arguments. sceKernelIsStack and the main thread's attributes report that range.
• An executable that does not load libkernel.prx keeps the stub as it was.

#3965 does the same on Windows, with a helper in libc that is built only there.

Title

Assassin's Creed Mirage (PPSA13960, 01.007.000) first stops on main at a kevent read event, which #6343 implements. With #6343 applied, it stops with SIGSEGV on the main thread 6 s after start, with nothing in the log.

• Its allocator finds the pool that owns a pointer through a table indexed by pointer >> 28 (eboot 0x570b9c1-0x570b9ec).
• The main thread asks it about an object on its own stack. 0x7fffffff42c8 gives index 0x7ffff, so the lookup reads past the table and dereferences what it finds there (0x8232f00).
• On the console the main thread's stack lies below 0x7eeffc000, which gives index 0x7e.

With this PR and #6343 it passes that point and goes through its start-up screens (photosensitivity warning, Ubisoft logo, intro video, disclaimer, autosave notice; screenshots from two runs) to the next unimplemented path:

AGC graphics: guest cube texture descriptor does not contain a multiple of 6 array slices (format 22, 128x128x7, type 11, tile 9, mips 0..0 of 1) (base array 0)

Tested

• PS5 (FW 13.60): a payload prints the address of a local in main, its pthread_attr_get_np stack range, a worker thread's range, and a malloc and an mmap address:

  MAIN|local=7eeffbcdc argv=7eeffbdb8 argc=1
  MAIN|attr init=0 get_np=0 getstack=0 stack=7eedfc000 size=0x200000 top=7eeffc000
  OTHER|malloc=880009c00 mmap=20001c000 main=200009310
  WORKER|local=7eedf7fbc
  WORKER|attr stack=7eede8000 size=0x10000 top=7eedf8000
  

• The runs below are on main 0d078f75d. Each title was relinked twice from the same ELF with the same options, with this PR's relinker: for A it is built with the libkernel.prx check forced off, so the stub is main's. Both runs of a title use this PR's runtime libraries.

• Assassin's Creed Mirage, with feat(libkernel): accept a kevent read event on descriptor 0 #6343's Kqueue.cpp change in libkernel:

  • A: SIGSEGV 6 s after start, as on main.
  • B: the cube texture error above, 120 s after start (a first run, which also compiles the title's shaders).

• Titles that already ran on main:

  Title	A	B
  Hades (PPSA03355, 01.003.000)	House of Hades, moved with the left stick	same
  CONTROL Resonant (PPSA34547, 01.300.300)	photosensitivity warning, end-user license agreement scrolled to its end, privacy policy	same
  Alan Wake 2 (PPSA02572, 01.200.007)	black screen that waits for a button, end-user license agreement	same
  ASTRO BOT (PPSA21567, 01.018.000)	title, save selection, galaxy map	same

  CONTROL Resonant's first A run went before B and showed its photosensitivity warning after the script's first press, so it ended one screen behind the script; the A row is a second A run made after B.

• guest_main_stack (new, Linux) enters a test entry through the helper and checks:

  • the argument;
  • that sceKernelIsStack reports a range ending at 0x7eeffc000 that holds the entry's locals;
  • the guard page below it;
  • that a guest thread created there gets a separate stack.

  With HostStackLimits as on main it fails at GuestMainStack.cpp:68.

• linux_main_stack_entry (new) relinks an executable that loads libkernel.prx and one that does not, and checks the stub and the imports of both.

  • On Linux x86-64 it also runs the first with a test libkernel.prx whose helper calls the entry point. The entry exits with its argc plus 41.
  • With the libkernel.prx check forced off it fails.
  • entry_stub checks both stubs.

• Linux (Arch, kernel 7.2.9, GCC 16.2.1, Release, AMD Radeon RX 9070 XT with RADV, Mesa 26.2.4): full CTest with -j8, 1439 of 1440 pass. agc_driver_tessellation segfaults the same way without this PR.

• Windows Server 2025 VM, WinLibs GCC 15.2, Release, BUILD_TESTING=ON, no GPU: full CTest, 1478 of 1479 pass (agc_driver_gated_dispatch_capture_short_timeout is skipped there). guest_positioned_close failed: it fails intermittently on main too, 5 of 30 runs of main 0d078f75d on the same VM. Windows does not use this stub.

• tools/check_conventions.py passes. tools/pr_overlap.py lists feat(relinker): support --trace-imports for Linux output #7417:

  • feat(relinker): support --trace-imports for Linux output #7417 adds TraceImport_nid_no_patch to the Linux output the same way (an undefined symbol with a GLOB_DAT slot) and puts a prologue before the entry stub, so the two change neighbouring lines of LinuxElfPatcher::Patch. With both symbols and both relocations kept and its prologue falling through to this stub, the merge builds and all 199 relinker tests pass, linux_main_stack_entry and linux_trace_imports included.
  • fix(libkernel): keep a protected host stack's guard page, and protect the Linux main stack #6443 also changes HostStackLimits: it caches the host stack range the function reads. The guest main stack check runs before that, so the two combine, and git merge-tree merges them without conflicts.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: PPSA13960 Assassin's Creed Mirage (PPSA13960) #7534
• GPUs tested: n/a
• AI-assisted: yes

🤖 Generated with Claude Code

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 2780	core/relinker/CMakeLists.txt
PR 3986	core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 4972	core/libs/prx/libkernel/CMakeLists.txt
PR 5021	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Pthread/src/Thread.cpp core/relinker/CMakeLists.txt docs/dev/debt/libkernel.md
PR 6044	core/libs/prx/libkernel/CMakeLists.txt core/relinker/CMakeLists.txt
PR 6149	core/relinker/elfpatcher/tests/GuestModuleArgsHost.cpp docs/dev/debt/libkernel.md
PR 6226	core/libs/prx/libkernel/CMakeLists.txt
PR 6446	docs/dev/debt/libkernel.md
PR 6469	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 6519	docs/dev/debt/libkernel.md
PR 6618	core/relinker/CMakeLists.txt
PR 6879	core/libs/prx/libkernel/Pthread/src/Thread.cpp core/relinker/CMakeLists.txt core/relinker/elfpatcher/include/elfpatcher/general/ElfConstants.hpp
PR 6908	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 6936	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 6942	core/libs/prx/libkernel/CMakeLists.txt
PR 6977	core/libs/prx/libkernel/CMakeLists.txt core/relinker/CMakeLists.txt docs/dev/debt/libkernel.md
PR 7039	docs/dev/debt/libkernel.md
PR 7049	docs/dev/debt/libkernel.md
PR 7075	docs/dev/debt/libkernel.md
PR 7088	docs/dev/debt/libkernel.md
PR 7105	docs/dev/debt/libkernel.md
PR 7121	docs/dev/debt/libkernel.md
PR 7129	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 7149	core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 7173	core/relinker/CMakeLists.txt
PR 7192	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 7201	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 7299	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 7332	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Pthread/src/Thread.cpp core/relinker/CMakeLists.txt docs/dev/debt/libkernel.md
PR 7417	core/relinker/CMakeLists.txt core/relinker/elfpatcher/src/linux/LinuxElfPatcher.cpp
PR 7453	core/relinker/CMakeLists.txt
PR 7509	docs/dev/debt/libkernel.md
PR 7594	core/relinker/CMakeLists.txt core/relinker/elfpatcher/src/general/EntryStubBuilder.cpp core/relinker/elfpatcher/tests/EntryStubTests.cpp

[oneandonlydean]

Thanks, the design looks right, and the stub, the 16-byte alignment and the bounds query all check out. One small safety change before merge: EnterGuestMainStack maps the stack with DoMapAnon(&mapping, bytes, 3, GUEST_MAP_FIXED) and exclusive = false, so ReplaceFixedOverlap silently replaces anything already registered in [top - bytes, top). With a large RLIMIT_STACK (up to 1 GiB) that range reaches well below the console stack. Nothing should be there at entry, but a collision would be clobbered rather than reported. Could it pass exclusive = true, so an overlap fails with ENOMEM and a named error?

On order: as discussed on #7594, this lands first and #7594 rebases with its MXCSR set before call [rip+slot].

[simonbbby]

I reproduced Dean's collision concern and tested the requested exclusive call locally on exact head 51e5eae15b432788ec554811a80c60ad74337e97. Ubuntu 24.04, kernel 7.0.0-38-generic, GCC 13.3, Release.

With soft RLIMIT_STACK=1 MiB, an external probe registers a 16 KiB RW anonymous guest mapping through DoMapAnon(..., exclusive=true), fills every dword with 0xa55a1234, then calls the real EnterGuestMainStack helper:

Existing sentinel page	Submitted head	Local exclusive-call proposal
Inside usable stack, at top - 0x10000	Enters; sentinel becomes zero	Refuses; all 16 KiB and RW protection preserved
Inside guard, at top - 0x104000	Enters; page protection becomes 0	Refuses; all 16 KiB and RW protection preserved
Immediately below mapping, at top - 0x108000	Enters; sentinel preserved	Same
No collision	Enters	Same

The local production change is only:

DoMapAnon(&mapping, bytes, 3, GUEST_MAP_FIXED, PS5_PAGE_SIZE, true)

Both refused cases report EnterGuestMainStack: cannot map the guest main stack before entry. This fits the mapping path: without exclusivity, the partial overlap reaches fixed-overlap replacement; with it, mutation.Overlaps returns ENOMEM before any replacement/protection change. The adjacent-page control rules out blanket refusal of nearby registered mappings.

Boundary/guard checks, on both the submitted head and the local proposal: an extension of GuestMainStack.cpp changes the soft limit after process startup and verifies these reported usable spans:

• 1 MiB → 1 MiB; 3 MiB + 1 byte → 3 MiB + 16 KiB; 8 MiB → 8 MiB; 64 MiB → 64 MiB.
• 2 GiB → 1 GiB; zero and unlimited → 8 MiB.

All 14 processes pass. Every reported upper bound is 0x7eeffc000; each run also forks a child with core dumps disabled/default SIGSEGV handling, and an actual read one byte below usable stack is terminated by SIGSEGV. The parent creates/joins a guest worker and keeps the original separate-stack assertions.

guest_main_stack, linux_main_stack_entry and entry_stub pass (3/3) with the proposal. Separately, restoring only the original HostStackLimits host-query behavior makes the stock test fail its upper-bound assertion at line 68. Restoring the exact PR source and rebuilding returns the three tests to passing.

I restored the submitted source; the worktree is clean. The exclusive call above is a tested local proposal, not a pushed change. Generated-input/runtime-helper checks only: no Mirage/title, Windows, ASan-offset or full-suite result.