fix(relinker): rewrite ALU reads of guest TLS through %fs in macOS output

[DavidSamir]

What

Morrowind (OpenMW, PPSA99630) cannot be converted for macOS on main:

FAIL: Unsupported macOS guest TLS instruction (bytes: 64 48 2b 0c 25 28 00 00 00) (offset 0x2ebbb8d)

64 48 2b 0c 25 28 00 00 00 is sub %fs:0x28,%rcx, the stack-protector check that compilers emit at a function's end: it subtracts the guard in the thread control block from the saved copy and branches on the flags. PS5MC (PPSA99420) stops on the same check (sub %fs:0x28,%rax, 64 48 2b 04 25 28 00 00 00) once #6410 lets it get past its first refusal. After this PR, Morrowind stops on its empty PT_TLS, which #6448 handles. The Windows TLS rewriter converts ALU reads of %fs:disp32 (add, or, adc, sbb, and, sub, xor, cmp into a 64-bit register); the macOS scan in MacOsImage.cpp accepts only loads and the stack-guard store, and refuses the module.

The macOS scan now accepts the same ALU reads as the Windows rewriter, and the writer gives each one a stub like a load's: it finds the guest thread pointer through the pthread key and reads the value at the displacement, then restores the guest's flags before applying the operation, so adc and sbb see the guest's carry and the operation's flags reach the branch after it. For rax the stub swaps the value with the saved rax on the stack and applies the operation to that memory operand. The displacement check of loads applies unchanged. Load and store stubs are byte-identical to main.

Four commits:

1. fix(relinker): rewrite ALU reads of guest TLS through %fs in macOS output.
2. test(relinker): rewrite stack-guard subtractions through %fs for macOS: a new macos_tls_alu test converts sub %fs:0x28,%rax and sub %fs:0x28,%rcx with --macos and checks that the access is gone and the stub applies the operation.
3. test(relinker): cover other ALU operations and registers reading guest TLS for macOS: xor into r10, adc from the TLS block (%fs:-8), cmp against the thread pointer (%fs:0).
4. test(relinker): keep refusing a macOS TLS ALU read outside the thread TLS block.

Tested

Windows host (4c4980e on main 0afa6a6, Windows 11 10.0.26300, WinLibs GCC 15.2.0 r7, Release). The relinker writes macOS output on any host:

• tools/check_conventions.py --base origin/main: passed. tools/pr_overlap.py: no overlap with open pull requests.
• With a relinker-only build (-DANYPS5_RELINKER_ONLY=ON -DBUILD_TESTING=ON), the relinker builds at each of the four commits and every test that runs passes. The 27 that fail are the same at every commit and come from that build: 25 test programs it does not compile, windows_diagnostic_fixture_isolation, which copies one of them, and check_main_sync, which reads git's UTF-8 output as cp1252 on this host (sp ace é.txt) and fails the same way on main. The change is in MacOsImage.cpp, which only the relinker links, so the relinker-only build runs every test that can see it.
• macos_tls_alu fails without the fix: with main's relinker, sub-rax-stack-guard stops with Unsupported macOS guest TLS instruction (bytes: 64 48 2b 04 25 28 00 00 00).
• The stubs were run in an emulator (Unicorn 2.1.4). For each form, the original instruction ran with the FS base at a guest thread pointer, and the converted Mach-O ran from the patched site to the next instruction with the pthread key slot and the GS table set up the way the stub reads them. Over 400 random register, flag, guard and TLS block values per form, every general register, RSP and the arithmetic flags (CF PF AF ZF SF OF) matched: sub rax and sub rcx against %fs:0x28, xor r10, %fs:0x28, adc rdx, %fs:-8, sbb rax, %fs:-8, cmp r8, %fs:0, and r15, %fs:-16, add rax, %fs:0, or rbx, %fs:0x28; a quarter of the trials set every register to the guard, so sub and cmp also produce zero. Two plain loads ran as controls and matched as well.
• Linux (WSL2 Ubuntu 24.04, GCC 13.3.0): the relinker builds and the macos|tls tests pass (guest_force_tls_destructor is not run because only the relinker target was built).
• macOS conversion of every catalog title (79), main's relinker against this branch's, same arguments: all 79 outputs are byte-identical, because neither title this unblocks converts with this PR alone. Morrowind (PPSA99630) gets past sub %fs:0x28,%rcx and stops on Guest TLS access without a non-empty PT_TLS (fix(relinker): convert stack-guard and thread-pointer reads with an empty PT_TLS for macOS #6448); PS5MC (PPSA99420) still stops first on the refusal fix(relinker): skip prefixed decodings inside an earlier instruction in the macOS TLS scan #6410 removes. With fix(relinker): skip prefixed decodings inside an earlier instruction in the macOS TLS scan #6410, this PR and fix(relinker): convert stack-guard and thread-pointer reads with an empty PT_TLS for macOS #6448 merged together, both convert, and 78 of the 79 titles convert (47 on main, whose 47 outputs are byte-identical); the 79th is Prospero Win (PPSA99995), whose GS instructions fix(relinker): trap guest GS accesses in macOS output instead of refusing the module #6458 handles.

Not run on macOS: I have no Mac. The stubs were executed in an emulator instead (above).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: macOS conversion of PPSA99630 (Morrowind, with fix(relinker): convert stack-guard and thread-pointer reads with an empty PT_TLS for macOS #6448) and PPSA99420 (PS5MC, with fix(relinker): skip prefixed decodings inside an earlier instruction in the macOS TLS scan #6410)
• AI-assisted: yes

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 2780	core/relinker/CMakeLists.txt
PR 5021	core/relinker/CMakeLists.txt
PR 6044	core/relinker/CMakeLists.txt
PR 6879	core/relinker/CMakeLists.txt
PR 6977	core/relinker/CMakeLists.txt
PR 7173	core/relinker/CMakeLists.txt
PR 7450	core/relinker/CMakeLists.txt core/relinker/elfpatcher/src/macos/MacOsImage.cpp
PR 7531	core/relinker/CMakeLists.txt
PR 7594	core/relinker/CMakeLists.txt

[jotitadev]

Ran this on a Mac, since the stubs were only executed in an emulator so far.

Machine: Apple M2 Max, macOS 27.2, AppleClang (Xcode 27). Head 4c4980e0b merged without a commit onto main 03c4cb5.

Suites (relinker-only Release)

• arm64: 174/174.
• x86_64 under Rosetta with /usr/bin/python3 3.9.6: 171/174. The 3 failures (pr_overlap_tools, release_package, linux_fixed_base) fail the same way on main with Python 3.9.
• macos_tls_alu with only the test applied to main: fails on both arches. With the fix: passes.

Stubs executed under Rosetta 2
I converted each form with --macos, mapped the output's __ELF* and __APS5* segments at their addresses in an x86_64 process (sysctl.proc_translated = 1), stored a real pthread_key_create key in the key slot the stub loads, and set its value with pthread_setspecific to a guest TCB whose word at offset 0 points to itself. A trampoline loaded all 15 GPRs and RFLAGS, called the patched site, and saved them back. The slow path was never reached (its callee is not bound in this harness), so every trial went through the %gs-indexed TSD lookup.

Forms, 1000 trials each with random registers, flags, guard and TLS words (a quarter set every register to the guard): sub rax/rcx, %fs:0x28, xor r10, %fs:0x28, adc rdx, %fs:-8, sbb rax, %fs:-8, cmp r8, %fs:0, and r15, %fs:-16, add rax, %fs:0, or rbx, %fs:0x28, sbb r11, %fs:0x28, adc rax, %fs:0x28.

Result: 11000/11000. The destination register matched the reference, every other GPR was unchanged, and CF PF AF ZF SF OF matched, with AF skipped for the logical ops. adc/sbb consumed the incoming carry correctly.

Not run: a real title. On macOS, main still can't link the runtime the output needs.

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[DavidSamir]

Hardware run on Apple silicon.

macOS 26.7.1, Apple M2 (8 GB), Rosetta 2, AppleClang 21.0.0. Relinkers built (-DANYPS5_RELINKER_ONLY=ON, Release) from main 5d9dcc4 alone and with this branch merged; catalog releases, sha256 checked, eboot and modules unwrapped with tools/unwrap_self.py, converted with --macos --to-rosetta. For the runs, libraries from a local tree of main ce991f1 with #4627, #4630, #6124, #6028, #6135 and mugurc's macOS branches merged, so they build on macOS; MoltenVK from the LunarG SDK 1.4.363.

• Morrowind (OpenMW, PPSA99630): the relinkers of main, fix(relinker): skip prefixed decodings inside an earlier instruction in the macOS TLS scan #6410, fix(relinker): convert stack-guard and thread-pointer reads with an empty PT_TLS for macOS #6448 and fix(relinker): trap guest GS accesses in macOS output instead of refusing the module #6458 fail with Unsupported macOS guest TLS instruction (bytes: 64 48 2b 0c 25 28 00 00 00) (offset 0x2ebbb8d). With this branch alone it gets past that read and stops on Guest TLS access without a non-empty PT_TLS, as the description says (it needs fix(relinker): convert stack-guard and thread-pointer reads with an empty PT_TLS for macOS #6448 too).
• With all four macOS relinker branches merged it converts, and at load stops on dyld: symbol not found in flat namespace '_ruyD9QkP8dI', which is fchown (feat(libkernel_sys): implement fchown #6411), the same import it is missing on Windows.

AI-assisted: yes