Skip to content

fix(libc): correct guest exception callback ABI on Windows - #1840

Merged
oneandonlydean merged 2 commits into
boykopovar:mainfrom
Prophet069:pr/guest-exception-abi
Oct 10, 2026
Merged

oneandonlydean merged 2 commits into
boykopovar:mainfrom
Prophet069:pr/guest-exception-abi

Conversation

@Prophet069

Copy link
Copy Markdown
Contributor

What

Correct two Windows calling-convention mismatches in libc's guest exception runtime. Guest calls pass the object pointer in RDI under System V, but the registered message-destructor lambdas and exception vtable entries previously used native Windows function pointers expecting RCX.

  • Register an explicit APS5_VABI message-destructor adapter for standard message throws and regex_error. The adapter calls the unchanged native destruction helper. The legacy destructor pointer storage and native/guest allocation markers remain unchanged.
  • Give guest what, non-deleting destructor and deleting destructor slots explicit APS5_VABI types and use guest adapters for all plain/message exception vtables. Each adapter calls the correctly declared native helper.

Normal Release previously happened to leave the same object in RDI and RCX, masking the registered-callback defect. Deliberately supplying a different object in RCX reproduced destruction of the wrong message; clearing RCX reproduced an invalid read. Virtual calls showed the same mismatch.

Object and RTTI layouts, message reference counting, existing malloc/free ownership, native host helpers and native exception routing are preserved. Deleting-destructor tests use standalone malloc-owned objects; they do not delete exceptions owned by the exception allocator. No new exception export aliases are added.

The two commits form one runtime ABI fix: the second extends the regression fixture introduced by the first. There is no dependency on the separate math or hardware-concurrency changes.

Potential overlap: #1551 also edits core/libs/prx/libc/CMakeLists.txt. Its precise_wait registration and this PR's MinGW exception-test block are in different current hunks. There is no functional dependency, but the shared file should be checked when either PR is merged.

Tested

Windows 10 x86-64, MinGW GCC 15.2, Ninja, Release, BUILD_TESTING=ON. Validation was repeated on this independent PR branch, head 05e1cf9d, based on upstream 04477932.

  • Built only libc, libkernel and the relevant regression targets in a separate build directory, with at most four parallel jobs. No full repository build.
  • Passed 27 CTests three consecutive times: 81/81 executions. This includes 12 virtual-call cases, 10 guest message-destructor cases, windows_exceptions, guest_thread_callback_abi, guest_exception_handler, guest_raise_exception and guest_unwind_module_info.
  • Covered mismatched and zero RCX, correct message/object selection, shared and final-release messages, typed guest catches, rethrow, cleanup and normal Release. A test-process-only wrapper around libc's native free import checks actual object/message frees and rejects duplicate or foreign frees.
  • Existing native exception, virtual what, cleanup, exception_ptr and cross-thread rethrow regressions passed. Windows disassembly confirms RDI object access and native-call register/shadow-space bridging.
  • Before the original fixes, four registered-message callback cases failed and 10/12 vtable cases failed. Plain non-deleting destructor cases had no observable failure because their helper ignores the object pointer. These earlier baseline results are distinct from the fresh PR-branch validation above.
  • git diff --check, the official conventions check against origin/main, and the test-registration check passed. The registration checker reports no newly unregistered executable targets.

Linux source compatibility was reviewed: APS5_VABI is empty there, native helper semantics are unchanged, and the Windows assembly/IAT regressions are MinGW-only. Linux build and runtime tests were not run because no usable Linux environment was available. These self-authored guest fixtures validate AnyPS5's runtime model, not proprietary PS5 SDK layouts or physical-console behavior.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes, with OpenAI Codex; the author reviewed the changes and remains responsible for them

Guest message throws registered native Windows callbacks even though Release invokes guest callbacks using System V. Register an APS5_VABI adapter in the standard and regex message throw paths and retain the native DestroyMessage helper and legacy callback storage type.

Add MinGW CTest regressions with distinct or zero RCX, shared and final-release messages, typed guest catches, rethrow and normal Release. All ten new cases plus Windows exception and thread callback regressions pass three repetitions. Linux execution remains unvalidated; guest vtables and host exception paths are unchanged.
Give guest exception what, non-deleting destructor and deleting destructor slots explicit APS5_VABI types and register guest adapters. Preserve native helpers, object layouts, native exception routing and existing malloc/free ownership.

Add mismatched and cleared RCX regressions for plain and message vtables, observing actual libc free calls to reject foreign or duplicate frees. Retain typed guest catch/rethrow and native exception coverage. All 27 targeted CTests pass three repetitions on Windows Release; Linux execution remains unvalidated.
oneandonlydean added a commit to oneandonlydean/AnyPS5 that referenced this pull request Oct 10, 2026
oneandonlydean added a commit to oneandonlydean/AnyPS5 that referenced this pull request Oct 10, 2026
@github-actions

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
#2545 core/libs/tests/GuestMessageDestructorAbi.S
core/libs/tests/GuestMessageDestructorAbi.cpp

@oneandonlydean
oneandonlydean merged commit 2061931 into boykopovar:main Oct 10, 2026
6 checks passed
SleglJan pushed a commit to SleglJan/AnyPS5 that referenced this pull request Oct 10, 2026
…-abi

fix(libc): correct guest exception callback ABI on Windows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants