fix(libkernel): handle fcntl and FIONREAD on a host socket descriptor

[milescoviello]

What

On Linux and macOS, fcntl and ioctl threw for a host socket that is not a guest socket:

fcntl: descriptor N is neither a file, a directory, a pipe nor a character device

A standard descriptor is such a socket when a service manager connects it to its journal, or when the process starts with a socket as stdin. That makes guest_kernel_fcntl abort in its StandardInputFlags case. Here the full ctest -j8 run on main 1e2d9b3 failed that test 3 runs out of 3, while the test passes on its own:

[libc] unhandled exception 'St13runtime_error' ...
  what():  fcntl: descriptor 0 is neither a file, a directory, a pipe nor a character device

Started with a socket as stdin from Python's subprocess, the same test binary aborts the same way every time; with /dev/null as stdin it passes.

A socket is now its own descriptor kind in DescriptorControl.cpp:

• Its status and close-on-exec flags go through the host, as for a pipe end.
• FIONREAD asks the host.
• Record locks and read-ahead hints fail with EBADF, as for a pipe. FreeBSD's kern_fcntl returns EBADF for these on a descriptor without a vnode.
• O_ASYNC still throws, as on a pipe.

Guest sockets keep their own path, and Windows classification is unchanged. The fcntl technical debt entry describes the new case.

Tested

Ubuntu 24.04, GCC 13.3, Release.

• guest_kernel_fcntl gains HostSocketFlags (not on Windows). It runs on a host socketpair:

  • F_GETFL reports O_RDWR;
  • F_SETFL with O_NONBLOCK reaches the host socket;
  • F_SETFD and F_GETFD round-trip;
  • FIONREAD reports the 3 queued bytes;
  • F_SETLK and F_RDAHEAD fail with EBADF;
  • O_ASYNC throws as on a pipe.

  Results:

  • With main's DescriptorControl.cpp the test aborts with fcntl: descriptor 5 is neither a file, a directory, a pipe nor a character device. With this PR it passes 3 of 3.
  • With a socket as stdin it now passes too.

• After rebasing onto main 4566cc3: the 419 guest_* tests pass. Before that rebase, the full suite passed 1437 of 1437 under ctest -j8, including guest_kernel_fcntl.

• Windows 11 (MinGW GCC 15.2) and Ubuntu 24.04 (GCC 13.3), this PR merged onto main 9f25d0b (run by @wachi35676). On Windows it builds and the socket and fcntl tests pass 38/38; the change and its test are for Linux and macOS only. On Linux, HostSocketFlags aborts with main's DescriptorControl.cpp and passes with this PR. guest_kernel_fcntl_tests started with a socketpair as stdin aborts on main and exits 0 with this PR.

• From the same run: for a datagram socket, Linux's FIONREAD counts only the next datagram, and FreeBSD's soo_ioctl counts every buffered byte. The fcntl entry in docs/dev/debt/libkernel.md now says so (91a9e56). The code still passes the host's count through.

• tools/check_conventions.py --base upstream/main passes.

• fix(libkernel): support socket fcntl duplication #7437 (guest sockets, F_DUPFD and F_DUP2FD on the libSceNet path) rewrites the same fcntl line in docs/dev/debt/libkernel.md. The code doesn't overlap: this PR only changes DescriptorControl.cpp for host sockets. Whichever PR lands second needs that sentence merged, and I'll rebase this one if it's second.

• @DavidAmsellem reproduced it and confirmed the fix on Linux Mint 22.3, AMD RX 7900 XT (RADV, Mesa 25.2.8). On plain main, guest_kernel_fcntl fails under ctest -j8 in 4 of 4 runs with the reported message, and starting the test with a socket-pair end as stdin aborts every time. With this PR (91a9e56 on 0bc3003), that start passes 3 of 3 and full ctest -j8 passes 1478 of 1480. The two failures happen on main too.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: AMD Radeon RX 7900 XT, RADV Mesa 25.2.8 (by @DavidAmsellem)
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 7437	docs/dev/debt/libkernel.md L199-200

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 5021	docs/dev/debt/libkernel.md
PR 6044	core/libs/tests/GuestKernelFcntl.cpp
PR 6149	docs/dev/debt/libkernel.md
PR 6446	docs/dev/debt/libkernel.md
PR 6469	docs/dev/debt/libkernel.md
PR 6519	docs/dev/debt/libkernel.md
PR 6740	docs/dev/debt/libkernel.md
PR 6936	docs/dev/debt/libkernel.md
PR 6977	core/libs/tests/GuestKernelFcntl.cpp docs/dev/debt/libkernel.md
PR 7049	docs/dev/debt/libkernel.md
PR 7075	docs/dev/debt/libkernel.md
PR 7088	docs/dev/debt/libkernel.md
PR 7105	docs/dev/debt/libkernel.md
PR 7121	core/libs/prx/libkernel/File/src/DescriptorControl.cpp docs/dev/debt/libkernel.md
PR 7149	docs/dev/debt/libkernel.md
PR 7332	core/libs/tests/GuestKernelFcntl.cpp docs/dev/debt/libkernel.md
PR 7509	docs/dev/debt/libkernel.md
PR 7541	docs/dev/debt/libkernel.md
PR 7554	docs/dev/debt/libkernel.md
PR 7556	docs/dev/debt/libkernel.md
PR 7605	docs/dev/debt/libkernel.md
PR 7653	docs/dev/debt/libkernel.md
PR 7937	docs/dev/debt/libkernel.md

[wachi35676]

Also built and ran on Windows 11 (MinGW GCC 15.2) and Ubuntu 24.04 (GCC 13.3), with this PR merged onto main 9f25d0bf5.

• Windows: it compiles and the socket/fcntl set passes 38/38. The change and its test are non-Windows by design.
• Linux: HostSocketFlags aborts with main's DescriptorControl.cpp and passes with this PR. guest_kernel_fcntl_tests started with a socketpair as stdin aborts on main ("descriptor 0 is neither…") and exits 0 with this PR.

One small difference you may want in the debt line: FIONREAD on a host socket passes the host's answer through. For a datagram socket, Linux reports only the next datagram: an AF_UNIX SOCK_DGRAM socket with 3+5 bytes queued gives 3. FreeBSD's soo_ioctl reports everything buffered, sbavail(&so->so_rcv) = 8 (sys_socket.c:202-205).

AI-assisted: yes (Claude Code); results above were run on this machine.

[milescoviello]

Thanks @wachi35676 for the Windows and Linux runs, and for the datagram FIONREAD case. I added both to the description. Linux gives no direct count of every queued datagram, so the code keeps passing the host's count through, and the fcntl debt entry now records the difference (91a9e56, with the soo_ioctl lines you linked).

AI-assisted: yes (Claude Code)

[DavidAmsellem]

Independently reproduced and confirmed fixed on another Linux machine.

• GPU/OS: AMD Radeon RX 7900 XT (RADV NAVI31, Mesa 25.2.8), Linux Mint 22.3 (Ubuntu 24.04 base),
  kernel 7.0.0-38, GCC 13.3
• Built 91a9e5608 merged into main 0bc3003f8, Release, BUILD_TESTING=ON

Before this PR, on plain main:

• ctest -j8 failed guest_kernel_fcntl with Subprocess aborted in 4 runs of 4, and in 3 more
  runs of a reduced set, while the test passes when run on its own.
• LastTest.log carries exactly the reported message:
  fcntl: descriptor 0 is neither a file, a directory, a pipe nor a character device.
• Starting the test binary with a socketpair end as stdin aborts every time; with /dev/null, a
  regular file, a pipe, or stdin closed, it passes. That matches the StandardInputFlags case
  calling fcntl(0, F_GETFL) on whatever descriptor 0 happens to be.

With this PR:

• The same socket-as-stdin start passes 3 runs of 3.
• Full ctest -j8 with ANYPS5_REQUIRE_VULKAN=1: 1478 of 1480, and guest_kernel_fcntl is no
  longer among the failures. The two remaining ones are unrelated to this PR and also fail on
  main here: agc_pipeline_cache_file (test(agc): measure pipeline cache hits against the bytes a pipeline adds #7676) and agc_driver_guest_memory_lifetime_direct
  (no read-write access to /dev/udmabuf on this machine).

AI-assisted (Claude Code).

[milescoviello]

Thanks @DavidAmsellem for reproducing it on AMD. The socket-as-stdin start is a clean way to show the cause. I added your runs to the description.

AI-assisted: yes (Claude Code)

[adriadam10]

Checked the diff against FreeBSD's kern_fcntl/soo_ioctl behaviour cited in the debt entry: Classify no longer throws for a host socket, every pipe-only branch on the non-Windows path (status flags, record lock, read-ahead, O_ASYNC, FIONREAD) now covers it, guest sockets and Windows are untouched. HostSocketFlags pins the values and throws on main; independent runs by @wachi35676 and @DavidAmsellem confirm it. The libkernel.md line only collides with #7437 by adjacency. Merged.