feat(relinker): frame red-zone leaves without a frame pointer by moving their rsp operands

[EagleBearerYS]

What

#5052's --frame-red-zone frames leaf functions with a push rbp; mov rbp, rsp prologue. Titles whose red-zone leaves have no frame pointer get no framing at all:

• Resident Evil Village (PPSA01556): 3,036 leaves keep values below RSP, 0 framed.
• ASTRO BOT (fdfawef324 on feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052): 455 leaves, 0 framed.
• GTA V (tibzejoker on feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052): 875 leaves, 0 framed.

In Village this is a boot crash, the #361 mechanism. A frameless merge leaf (app.exe+0x1469cbb..) keeps its loop state at -0x8 to -0x28(%rsp) and copies 48-byte entries with vmovups. One of its stores faulted on a write-tracked view and was resumed. Afterwards the slots below RSP held zeros and YMM upper halves. The loop stopped short and left zeroed entries, and GameThread later read 0x18 from a null pointer at app.exe+0x69cb959. That happened in 2 of 5 boots.

This builds on #5052. Only the last two commits are this pull request:

• Leaves without a frame pointer (743ce53):
  • What gets framed. A leaf that starts with a chain of pushes of at least five bytes, and leaves through pop ...; ret exits of at least five bytes, gets the same prologue and exit stubs as feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052's leaves.
  • The operands. Each of its RSP operands is moved by the 128 bytes RSP is lowered by. A negative 8-bit displacement stays 8-bit, and a lea of a slot below RSP moves the same way, so the address it computes is unchanged.
  • What's skipped. A leaf is skipped when an operand can't move in place: one at or above RSP with an 8-bit displacement, or one without a displacement. It is also skipped when it copies RSP into another register (the one-byte, two-byte and VEX forms, such as cmov and shlx), and in every case feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052 skips.
• Where a leaf's body ends (a8ed810). This is the case oneandonlydean raised on feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052, and it applies here with more force.
  • The problem. A leaf's body ran to the next known function start or int3 padding. A function reached only through a pointer, placed right after a leaf with no int3 between them, was framed as part of the leaf. Its RSP operands were moved and its exits raised RSP, although nothing had lowered it.
  • In Village: that happened once. The function at 0x1693a0 follows the ret of the leaf at 0x1691f0 with no padding, and is never called directly.
  • The fix. The body now ends at 16-byte-aligned code after a ret or ud2, and at the target of a jmp when only padding lies between them (a tail call to the next function). Code past such a point is framed only from a real function start.
  • Why not after every jmp. Aligned code after any other jmp stays in the body, because clang aligns rotated loop headers there. Cutting after every jmp skipped 2,129 of Village's 2,131 framed leaves.
  • What's left, in the debt entry: a pointer-only leaf right after a jmp to elsewhere, such as a loop's back edge, without int3 padding. Village has none (see Tested). If feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052 takes its bounds from the .eh_frame FDEs, as its review asks, this pull request uses them on rebase.

The debt entry and USAGE.md describe the new shape. Inside a moved frame, the CFA rules of the guest's unwind tables are 128 bytes off; the debt entry records it.

#5596, tibzejoker's port of #637, avoids the write fault itself for direct memory mapped at one address. Framing protects a leaf from any exception frame Windows writes below RSP while the leaf runs, whatever memory faulted, so the two cover different cases.

Tested

• Windows 11, WinLibs GCC 15.2.0 r7, Release, on feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052's head (fab7d20): red_zone_framer.
  • New cases:
    • a leaf with a frame pointer keeps its operands;
    • a leaf that only pushes is framed: its prologue site covers the pushes, its exit site covers the pops and ret, and both its RSP operands are moved;
    • leaves whose RSP operands can't all move, or whose prologue is too short, are skipped with their reasons;
    • a leaf that copies RSP into another register is skipped;
    • a leaf reached without a call, right after another leaf, isn't framed as part of it;
    • a leaf that tail-calls the function after it is skipped;
    • a leaf whose loop header is aligned after a jmp is still framed.
  • With frameless leaves skipped, as on feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052's head, the frameless cases fail ("a leaf that only pushes was not framed").
  • The pointer-only case fails with the first commit and passes with the second.
• GitHub's Linux and Windows runners, on my fork: red_zone_framer passes 20 of 20 runs. With the first commit's framer, the second commit's case fails (https://github.com/EagleBearerYS/AnyPS5/actions/runs/38498470455).
• Full Windows and Linux builds and tests on my fork, with this branch merged into main feddf7f:
  • Windows: 1110 of 1110 pass, with the Vulkan tests skipped because the runner has no Vulkan device.
  • Linux: 1090 of 1090 pass.
  • The relinker builds on Ubuntu, Windows and macOS.
  • The latest run, with both commits, is https://github.com/EagleBearerYS/AnyPS5/actions/runs/38498472031.
• Resident Evil Village (PPSA01556, 01.210.000) on Windows 11 26300, Ryzen 9 5900HS (Zen 3, no AVX-512), RTX 3070 Laptop GPU, on a local branch past the title's other stops (Resident Evil Village (PPSA01556) #5169):
  • The relink with --frame-red-zone:
    • 3,175 red-zone leaves, 2,132 framed, 14,080 RSP operands moved, 2,132 exits;
    • skipped: an exit shorter than a jump 498, an RSP operand that can't move 337, another use of RSP 179, a branch out of the function 17, an indirect jump 10, a prologue too short 2;
    • with feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052 alone: 3,036 red-zone leaves, 0 framed.
  • Against the unwind table (304,570 FDEs, read through PT_GNU_EH_FRAME, comparing the framed and unframed executables):
    • every rewritten function is framed at its own FDE start;
    • no function has rewritten bytes without its start being framed, and no rewritten byte lies outside every FDE;
    • before the second commit, 0x1693a0 was the one exception.
  • The boot crash: 2 of 5 boots without framing. With framing, 0 of 12: 10 at the first commit and 2 at the second.
• python3 tools/check_conventions.py --base fab7d203f (feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052's head): passed.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows #5052
• Related: Windows: resuming a write fault overwrites the guest red zone when AVX state is live #361
• Unblocks: PPSA01556 Resident Evil Village (PPSA01556) #5169 (boot crash in 2 of 5 boots)
• GPUs tested: NVIDIA GeForce RTX 3070 Laptop GPU, Windows 11
• AI-assisted: yes

[adriadam10]

Thanks for the detailed write-up. Two blockers before this can be reviewed:

• Base PR is closed. Depends on: #5052 was closed unmerged on 2026-10-14, so this branch still carries its whole RedZoneFramer (the diff against main is +659) plus Josip Parlov's commit. Either coordinate with them to reopen/land it, or resubmit as a self-contained PR with the authorship preserved.
• Motivation has changed. #361 is now fixed at the source by #5596 (merged). On #5052 parl0v measured 0 crashes in 20 boots of PPSA25344 without the flag on main with #5596. Please re-measure the Village boot crash (2 of 5 without framing) on current main with #5596 included, with and without --frame-red-zone. If it no longer reproduces, there is nothing for this opt-in pass to fix.

If it still reproduces, also note that #5052's review asked for function bounds from the .eh_frame FDEs (EhFrameReader) rather than heuristics; the "end of body" rule added in a8ed810 (aligned code after ret/ud2, jmp to padding) is a heuristic that leaves a documented gap, so it should use the FDEs here too.

[EagleBearerYS]

Re-measured on current main with #5596: Resident Evil Village (PPSA01556, 01.210.000), Windows 11 Home 10.0.26300, RTX 3070 Laptop (driver 617.42), unframed (relinked with main's relinker).

• On our stack from before fix(libc): track writes to singly mapped direct memory without faults on Windows #5596, stop 21 (Windows: resuming a write fault overwrites the guest red zone when AVX state is live #361) hit 35-52 s into a boot, as a GameThread access violation, a null read at app.exe+0x69cb959, in 2 of 5 boots.
• On main with fix(libc): track writes to singly mapped direct memory without faults on Windows #5596 it doesn't reproduce, 0 of 17 boots:
  • 10 boots to 120 s, back to back, on main 23be5a0 + fix(agc): heap-mirror ranges whose import windows left the write watch #5688 + fix(libkernel): commit a reserved host stack before protecting it #5703 + fix(agc): bind a read-only V# over the cached address space through the region that holds it #5815. None faulted, and several had reached the start-up autosave notice by then.
  • 7 earlier boots that ran past that window, 95 s to 32 min each, on main at 0b6d033, eac1e26 and 23be5a0, six of them with fix(agc): heap-mirror ranges whose import windows left the write watch #5688 and fix(libkernel): commit a reserved host stack before protecting it #5703.
• Both sets carry local commits that prepare shaders at their first draw (APS5_RE8_LAZY_PREPARE=1, with APS5_NO_GPL=1). Most also have local fixes for later stops.

So there's nothing left for this pass to fix in Village, and I'm closing it. Thanks for the review. parl0v's .eh_frame version (parl0v/AnyPS5@3ff74b9) is the starting point if a title needs framing again.