feat(libkernel): pass descriptors in SCM_RIGHTS messages on Unix sockets

[milescoviello]

What

sendmsg threw control data is not supported for any control buffer. Wine passes descriptors to and from its wineserver this way, so Forever Launcher's in-process Wine (PPSA99030) stops on it. On a guest Unix-domain socket, SCM_RIGHTS control messages now pass descriptors, as FreeBSD 11's unp_internalize and unp_externalize do.

• sendmsg: reads FreeBSD's layout: a 12-byte cmsghdr with the descriptors at offset 16, and records CMSG_SPACE bytes apart. It sends the descriptors with the host's SCM_RIGHTS, because a guest descriptor here is a host descriptor (a guest socket's number is a duplicate of its host socket).
  • It returns EBADF for a descriptor that is not open, and EINVAL for a level other than SOL_SOCKET or a record longer than the buffer.
• recvmsg with a control buffer: receives with the host's SCM_RIGHTS. A received socket is installed as a guest socket (family and type from SO_DOMAIN/SO_TYPE); any other descriptor is used as it is.
  • It writes FreeBSD's record, CMSG_LEN in cmsg_len and CMSG_SPACE bytes, truncated to the caller's buffer with MSG_CTRUNC, as kern_recvit copies it.
  • MSG_CMSG_CLOEXEC marks the received descriptors close-on-exec.
• Still throws: Windows (its socket pairs are loopback TCP), control messages other than SCM_RIGHTS, control data on IP sockets, and MSG_PEEK with a control buffer. The debt entry says so.

Tested

Ubuntu 24.04, GCC 13.3, Release, on main 673f740. The PR is rebased onto 864729d, whose two newer commits touch neither libkernel nor its tests.

• New guest_socket_rights test (Linux), over a stream socket pair:
  • It passes a pipe's write end, writes through the received descriptor and reads it from the pipe. The received descriptor is not close-on-exec.
  • It passes the ends of a stream and a datagram socket pair in one record with MSG_CMSG_CLOEXEC, and checks that both received sockets are close-on-exec and send and receive.
  • It passes another pipe end with MSG_CMSG_CLOEXEC and checks that the received host descriptor is close-on-exec. A pipe end goes through unchanged, so this reads the host flag, while the sockets' check reads the guest's close-on-exec set.
  • A control buffer of 16 bytes gets the header and MSG_CTRUNC, and a receive without a control buffer gets the data only.
  • A closed descriptor gives EBADF, and another level or an overlong record gives EINVAL.
  • With main's Udp.cpp it aborts on sendmsg: control data is not supported. It passes 3 of 3, with guest_socket_pair too.
• ctest -R '^guest_|libc': 402 of 402 pass.
• Forever Launcher (PPSA99030, 01.000.012), in gdb: Wine's first such call is sendmsg on a socket-pair end with MSG_NOSIGNAL | MSG_DONTWAIT, one byte of data and a 24-byte control buffer holding one SOL_SOCKET/SCM_RIGHTS record (cmsg_len 20) that passes the end of another socket pair.
  • On main with the PRs for its earlier stops merged locally (feat(libkernel): declare physhm_open and physhm_unlink #6915, fix(libkernel): export sysarch so modules that import it load #6028, fix(libSceSystemService): declare the PsmKit local process exports #6939, feat(libSceKeyboard): export missing firmware functions from database #6787, feat(libkernel): report the guest address-space map through kern.proc.vmmap #7070, fix(memory): honor low guest mapping hints #6469), it stops at 0 s on sendmsg: control data is not supported, 2 of 2.
  • With this PR it gets past it 2 of 2: the descriptor reaches the wineserver thread, and the title stops 1 s in at its next unimplemented call, kevent: NOTE_LOWAT (wineserver's kqueue registers EVFILT_READ and EVFILT_WRITE with NOTE_LOWAT 1). That is a separate PR.
  • Since fix(libkernel): take NOTE_LOWAT on socket read and write events #7092 (NOTE_LOWAT) merged, that next stop is gone. On main 98c251f with feat(libkernel): declare physhm_open and physhm_unlink #6915, fix(libSceSystemService): declare the PsmKit local process exports #6939, feat(libSceKeyboard): export missing firmware functions from database #6787, feat(libkernel): report the guest address-space map through kern.proc.vmmap #7070, fix(memory): honor low guest mapping hints #6469 and this PR (17d9e0a), Forever Launcher runs the whole 60 s in 2 of 2 runs with no unimplemented call. Its first-start prefix setup then reports title wineboot skipped; prefix must be initialized by payload Wine and failed to open "C:\\windows\\system32\\wineboot.exe", and Wine exits with 0xC0000135, the launcher's "Setting up the Windows files did not finish". I read that as the launcher's own setup, not a missing emulator call, but I haven't looked further.
• Windows build: @wachi35676 found that Udp.cpp did not compile there: GuestControlCloseOnExec was declared inside the non-Windows block. 17d9e0a moves the declaration out of it. A MinGW-w64 -fsyntax-only compile of Udp.cpp stops on that line before and passes after, and the 11 socket tests and 411 ^guest_ tests pass again on Linux. @screamm then built head 17d9e0a516 on Windows (Windows 11, WinLibs GCC 15.2.0, full build and libs). Moving the declaration back reproduces the compile error, and the 35 Windows socket, poll, select, rudp, equeue and kqueue tests pass 3 of 3. On Windows, sendmsg with an SCM_RIGHTS buffer still stops with control data is not supported on Windows, as this PR intends. None of their local titles sends or receives such a message.
• macOS: @jotitadev found that SO_DOMAIN and MSG_CMSG_CLOEXEC don't exist in XNU's headers, so Udp.cpp didn't compile on top of feat: macOS on Apple Silicon host support (Rosetta 2 and MoltenVK) #7332. aff8c2a takes the received socket's family from getsockname(). Where MSG_CMSG_CLOEXEC doesn't exist, it marks each received descriptor with fcntl(F_SETFD, FD_CLOEXEC) when the guest asked for it. Their local equivalent compiled with -Wall -Wextra on arm64 and x86_64, and guest_socket_rights passed 5 of 5 under Rosetta. They also found that the test didn't check the host flag, so a3b3801 adds the pipe-end case above. On Linux the head passes 3 of 3, and so does a build with the receive flag forced to 0, through the fcntl path. With the flag forced to 0 and the fcntl loop removed, the test fails. ^guest_|libc passed 411 of 411 on aff8c2a.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: PPSA99030 (Forever Launcher), together with feat(libkernel): declare physhm_open and physhm_unlink #6915, fix(libkernel): export sysarch so modules that import it load #6028, fix(libSceSystemService): declare the PsmKit local process exports #6939, feat(libSceKeyboard): export missing firmware functions from database #6787, feat(libkernel): report the guest address-space map through kern.proc.vmmap #7070 and fix(memory): honor low guest mapping hints #6469; no bring-up discussion exists yet
• GPUs tested: n/a
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 7675	core/libs/prx/libkernel/Socket/src/Udp.cpp L1368

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 4972	core/libs/prx/libkernel/CMakeLists.txt
PR 5021	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 6044	core/libs/prx/libkernel/CMakeLists.txt
PR 6149	docs/dev/debt/libkernel.md
PR 6226	core/libs/prx/libkernel/CMakeLists.txt
PR 6446	docs/dev/debt/libkernel.md
PR 6510	docs/dev/debt/libkernel.md
PR 6519	docs/dev/debt/libkernel.md
PR 6534	docs/dev/debt/libkernel.md
PR 6908	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 6936	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 6942	core/libs/prx/libkernel/CMakeLists.txt
PR 6977	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 7039	docs/dev/debt/libkernel.md
PR 7049	docs/dev/debt/libkernel.md
PR 7075	core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 7088	core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 7105	docs/dev/debt/libkernel.md
PR 7121	docs/dev/debt/libkernel.md
PR 7129	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 7149	docs/dev/debt/libkernel.md
PR 7192	core/libs/prx/libkernel/CMakeLists.txt docs/dev/debt/libkernel.md
PR 7201	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7332	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 7450	core/libs/prx/libkernel/CMakeLists.txt
PR 7509	docs/dev/debt/libkernel.md
PR 7565	docs/dev/debt/libkernel.md
PR 7653	docs/dev/debt/libkernel.md

[wachi35676]

This doesn't compile on Windows (head 4cd70d4ae; WinLibs GCC 15.2.0, Ninja, Release), alone or merged onto main 11acfdd84:

core/libs/prx/libkernel/Socket/src/Udp.cpp:1371:51: error: 'GuestControlCloseOnExec' was not declared in this scope

GuestControlCloseOnExec is declared at line 1244, inside the #ifndef _WIN32 block that runs from line 1240 to 1328. The flag check at line 1371 in recvmsg_nid_postfix, (passing ? GuestControlCloseOnExec : 0), is outside any guard. The CI windows job was skipped for this PR, so it didn't show. Declaring the constant above the #ifndef _WIN32 fixes the build here.

AI-assisted: yes (Claude Code); results above were run on this machine.

[milescoviello]

Thanks @wachi35676, that's right. 17d9e0a declares GuestControlCloseOnExec above the #ifndef _WIN32 block, as you suggested; recvmsg masks the flag on every host.

Checked here with MinGW-w64 (x86_64-w64-mingw32-g++-posix): a -fsyntax-only compile of Udp.cpp stops on exactly your line at 4cd70d4, and passes with 17d9e0a under -Wall -Wextra with no new warnings. On Linux (GCC 13.3, Release) the 11 socket tests pass 3 of 3, and the 411 ^guest_ tests pass. I haven't run a full Windows build.

AI-assisted: yes (Claude Code)

[screamm]

Windows build of head 17d9e0a51, merged onto main 0d078f75d together with #7492 (fd2eb3521) and #7509 (c170bba42); all three merge cleanly.

• Host: Windows 11 Pro 26220, Ryzen 7 5700X, RTX 4070 (driver 596.21), WinLibs MinGW-w64 GCC 15.2.0 r7, Ninja, Release, BUILD_TESTING=ON.
• A real full build and libs both pass (not a syntax-only compile). Udp.cpp builds with no warning, and g++ -Wall -Wextra -fsyntax-only of it with the project's flags is clean.
• Control for the fix: moving GuestControlCloseOnExec back inside #ifndef _WIN32 reproduces error: 'GuestControlCloseOnExec' was not declared in this scope at the recvmsg flag check (1391:51 in my copy, the same expression @wachi35676 reported at 1371:51); with the declaration above the block it compiles.
• guest_socket_rights is registered only if(NOT WIN32), so on Windows it neither runs nor skips; ctest does not list it. The 35 Windows socket, poll, select, rudp, equeue and kqueue tests pass 3x on a quiet host (105/105).
• Windows behaviour with a scratch probe (not part of the PR): on a socketpair, sendmsg without control and recvmsg with a control buffer behave as before (data, controlLength 0, flags 0), recvmsg with MSG_CMSG_CLOEXEC still fails with 45, and sendmsg with a one-descriptor SCM_RIGHTS buffer ends in terminate with sendmsg: control data is not supported on Windows. With Udp.cpp from main the probe differs only in the message text, so the 32 socket tests cannot tell the two apart on Windows.
• Non-GPU suite (ctest -j8 --timeout 120 -E '^agc_driver_|^video_out_flip'): 936 tests, 929 pass, 2 skipped, 5 load timeouts that pass serially. tools/check_conventions.py --base origin/main and git diff origin/main --check pass.

Titles: PPSA99030 is not available here. recvmsg is imported by 13 and sendmsg by 12 local titles. I ran PPSA74126 (Stremio Plus) and PPSA99082 (OpenNOW) for 45 s each, and PPSA99169 (RetroArch, app.exe only; its picker child's output is not captured), with a local diagnostic libkernel (reverted) that logged every sendmsg and recvmsg call with a control buffer. It logged none, so I have no local title that sends or receives an SCM_RIGHTS message.

Not checked: the descriptor passing itself (Linux only, guest_socket_rights not run), Linux, a console, PPSA99030.

Written with Claude Code assistance (validation by a subagent, reviewed in this session).

[milescoviello]

Thanks @screamm for the full Windows build, the control and the probe. I added it to the description.

AI-assisted: yes (Claude Code)

[jotitadev]

macOS build note on head 17d9e0a51.

The new code sits under #ifndef _WIN32, so it is also compiled on macOS hosts, but two of its identifiers do not exist in XNU's headers:

• core/libs/prx/libkernel/Socket/src/Udp.cpp:1315: getsockopt(native, SOL_SOCKET, SO_DOMAIN, ...). macOS has no SO_DOMAIN.
• core/libs/prx/libkernel/Socket/src/Udp.cpp:1396: MSG_CMSG_CLOEXEC. macOS has no such recv flag.

Repro on an M2 Max (macOS 27.2, Apple clang 21.0.0, arm64): merging this head onto #7332 (b182750ca, the PR that builds libkernel on macOS) is conflict-free. A -fsyntax-only compile of the merged Udp.cpp then fails with exactly these two errors:

Udp.cpp:1421:42: error: use of undeclared identifier 'SO_DOMAIN'
Udp.cpp:1502:122: error: use of undeclared identifier 'MSG_CMSG_CLOEXEC'

The same file from #7332 alone compiles cleanly. The line numbers above are in the merged file; the lines in this PR are 1315 and 1396. guest_socket_rights is registered under if(NOT WIN32), so it would be built on macOS too.

Suggestion that keeps one path for both Unix hosts:

• Take the family from getsockname() (sockaddr_storage::ss_family) instead of SO_DOMAIN. On macOS this returns AF_UNIX for a socketpair end and AF_INET6 for an unbound IPv6 socket, and SO_TYPE works as it is.
• Drop MSG_CMSG_CLOEXEC and set FD_CLOEXEC with fcntl on each received descriptor when the guest asked for it, before InstallPassed. Alternatively, guard the flag with #ifdef MSG_CMSG_CLOEXEC and use the fcntl path otherwise.

Not checked: running guest_socket_rights on macOS. The full libs build on macOS needs #7332 and an x86-64 build, which I could not run here.

AI-assisted: yes (Claude Code)

[milescoviello]

@jotitadev thanks, both are right. I took your first suggestion and the #ifdef variant of the second, in aff8c2a:

• The received socket's family comes from getsockname() (ss_family) instead of SO_DOMAIN. SO_TYPE is unchanged.
• Where MSG_CMSG_CLOEXEC exists, recvmsg still sets close-on-exec atomically, so Linux behaves exactly as tested. Elsewhere each received descriptor gets fcntl(F_SETFD, FD_CLOEXEC) right after recvmsg when the guest asked for it, before InstallPassed. I kept the atomic flag where it exists because the fcntl path leaves a window in which another thread's spawn could inherit the descriptor.

I'm running guest_socket_rights on Linux, plus once with the fallback forced so the fcntl path runs too, and the ^guest_|libc set. I'll push when those pass. I can't build for macOS here, so a -fsyntax-only check of the pushed head on top of #7332 would be welcome if you have time.

AI-assisted: yes (Claude Code)

[jotitadev]

@milescoviello thanks. feat/scm-rights is still at 17d9e0a51, and aff8c2a isn't on GitHub yet, so I checked the current head plus a local equivalent of what you described. I'll re-run on the pushed head.

M2 Max, macOS 27.2, Apple clang 21.0.0. Base: #7332 at 4a1c11a91 (the PR merges onto it without conflicts).

• 17d9e0a51 on feat: macOS on Apple Silicon host support (Rosetta 2 and MoltenVK) #7332: Udp.cpp still fails -fsyntax-only for arm64 and x86_64, with the same two errors (SO_DOMAIN, MSG_CMSG_CLOEXEC). feat: macOS on Apple Silicon host support (Rosetta 2 and MoltenVK) #7332 alone compiles.
• Local equivalent: family from getsockname() ss_family, MSG_CMSG_CLOEXEC under #ifdef, and otherwise fcntl(F_SETFD, FD_CLOEXEC) on each received descriptor before InstallPassed. Udp.cpp compiles with -Wall -Wextra and no warnings on both arches. In a real x86_64 build run under Rosetta, guest_socket_rights passes 5 of 5. The 20 guest_socket_*, guest_udp, guest_rudp_*, guest_poll and guest_select* tests pass. In 3 repeats, guest_socket_read_write failed once and passed on re-run; it is already intermittent on feat: macOS on Apple Silicon host support (Rosetta 2 and MoltenVK) #7332 alone.

One gap in the test: it never checks close-on-exec on the received host descriptor. The only ControlCloseOnExec case receives two sockets. For a guest socket, F_GETFD reads closeOnExecDescriptors, which InstallPassed fills, and not the host fd. So the check passes when nothing sets the host flag:

• macOS, local equivalent with the fcntl loop removed: guest_socket_rights passes 3 of 3.
• Linux (Ubuntu 26.04, GCC 15.2, Release), 17d9e0a51 with MSG_CMSG_CLOEXEC removed from the recvmsg call: it also passes 3 of 3.

So your forced-fallback run on Linux will pass whether or not the fcntl path runs. One way to cover both paths is to receive a passed pipe end with ControlCloseOnExec and require fcntl_nid_postfix(writer, GetDescriptorFlags) == 1. A pipe goes through InstallPassed unchanged, so the guest sees the host flag. With that check, both mutations fail (passed pipe without FD_CLOEXEC), and the unmodified code passes on macOS (with the fallback) and on Linux.

Not checked: your actual aff8c2a, arm64 runtime (the libs build is x86-64 only on macOS).

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[milescoviello]

@jotitadev thanks, and sorry the commit wasn't up yet when you looked. Both are pushed now:

• aff8c2a is the fix as described: getsockname() for the family, MSG_CMSG_CLOEXEC where it exists, otherwise fcntl(F_SETFD, FD_CLOEXEC) per received descriptor.
• a3b3801 is the check you suggested. CheckPassedPipe now also passes a pipe end with ControlCloseOnExec and requires F_GETFD == 1 on it.

You're right that my forced-fallback run proved nothing before that check. With it, on Linux (Ubuntu 24.04, GCC 13.3):

• The head passes 3 of 3.
• With the receive flag forced to 0, it passes through the fcntl path.
• With the flag forced to 0 and the fcntl loop removed, it fails.
• The ^guest_|libc set passed 411 of 411 on aff8c2a.

A run of -fsyntax-only on a3b3801 over #7332 would confirm the macOS side whenever you get to it.

AI-assisted: yes (Claude Code)