fix(libkernel): fail a malformed ioctl request with ENOTTY

[Lindamea1]

What

On main, ioctl and _ioctl throw when a file, pipe or character device gets a request word that FreeBSD's sys_ioctl rejects as malformed, unless the word is in the termios group t (such a word already fails there with ENOTTY); a closed descriptor fails with EBADF instead of ENOTTY, and a guest socket given a null argument with EFAULT. The OpenOrbis PS4 toolchain's net_http sample stops there: its musl sends the Linux TIOCGWINSZ (0x5413, no direction bit) to standard output on the first printf, and the run ends with

ioctl: request 0x00005413 on a file is not implemented, fd=1

@Get-fix reported this on Windows 11 in #5564; with #5569 the sample ran its whole flow. #3093 tracks it as [ ] Linux terminal ioctl 0x5413 (#5564, #5569), and its later updates (1, 2) count a change that answers the Linux TIOCGWINSZ with ENOTTY among the changes that let 14, then 15, of the 16 samples start on Windows. The other changes listed there (#5990 to #5994, #6376, #5138) are merged, and Get-fix's open #6519, #6520 and #6521 were tested with such a change too. On Linux, main's libkernel.prx throws the same line for ioctl(1, 0x5413, buffer) when standard output is redirected to a file (probe under Tested).

This carries @husseinkad's commit from #5569 (opened by @MarcZX10), with his authorship kept (git cherry-pick -x 3eb7edd9cf). #5569 was closed with "Closing: no title is known to reach this. [...] If a title stops on this on current main, reopen with the title ID and the log line." The log line is above; I can't reopen #5569, so this is a new pull request. Its two conflicts with current main are resolved without changing its code: the test keeps main's CloseOnExecRequest and KeepOnExecRequest next to MalformedRequests, and the debt entry keeps main's FIOCLEX wording with the commit's sentence added.

Reference: FreeBSD 11.4 sys_ioctl truncates the request to 32 bits and returns ENOTTY if it has none of IOC_VOID, IOC_IN and IOC_OUT, if it has IOC_OUT and length 0, or if it has IOC_VOID and a length that is neither 0 nor sizeof(int) (encoding). That comes before the argument's copyin and before kern_ioctl looks up the descriptor, so it wins over EFAULT and EBADF. The IOC_OUT test is the branch built with any of COMPAT_FREEBSD4, COMPAT_FREEBSD5 and COMPAT_43, and amd64 GENERIC has the first two; without them, IOC_IN with length 0 fails as well.

• File::IoctlRequestMalformed applies those three tests. File::Ioctl checks it first, and ioctl_nid_postfix checks it for a guest socket before the lookup and the null-argument EFAULT, so every descriptor kind, a closed one included, fails with ENOTTY.
• Well-formed requests take the same path as before.
• My commit adds 0x5413 to the test's malformed words and pins the two boundary words the check takes as well-formed: 0x20046601 (IOC_VOID, length 4) and 0x80006601 (IOC_IN, length 0) given to a guest socket with a null argument fail with EFAULT, not ENOTTY. It also rewrites the carried debt sentence as the open gap: an input request with length 0 is not rejected as malformed, and which of the COMPAT options the PS5 kernel has is unknown.

Differences from the reference: an IOC_IN request with length 0 is still treated as well-formed, as in a kernel with those COMPAT options; whether the PS5 kernel has them is unknown, as docs/dev/debt/libkernel.md records. Not measured on a console.

Tested

• Gentoo Linux, kernel 7.2.8, GCC 16.2.1, glibc 2.43, CMake 4.3.4, Ninja 1.13.2; Release build with BUILD_TESTING=ON on upstream main 43a2abaf7.
• guest_kernel_fcntl (core/libs/tests/GuestKernelFcntl.cpp) sends 0, 123, 0x5401, 0x5413, 0x40007468, 0x20087468 and 0x00087468, each with a buffer and with a null argument, to an open UDP socket, an open file, the same file after close, descriptor -1 and a pipe, and requires -1 with ENOTTY. It also sends 0x20046601 and 0x80006601 with a null argument to the socket and requires -1 with EFAULT. Its existing checks of FIOCLEX, FIONCLEX, FIONBIO, FIONREAD, the BSD TIOCGWINSZ (ENOTTY) and FIOASYNC (throws) are unchanged. It passed 3 runs in a row.
• Negative control, with this branch's test: with DescriptorControl.cpp, DescriptorControl.hpp and Udp.cpp from main, guest_kernel_fcntl aborts at its first null-argument call on the socket, ioctl(socket, 0, nullptr), which fails with EFAULT (guest errno 14 at the abort, under gdb). With only Udp.cpp from main it aborts at the same call. With all three from main, a scratch copy of the test that runs one section per process shows the file and pipe sections ending with terminate called after throwing an instance of 'std::runtime_error' and what(): ioctl: request 0x00000000 on a file is not implemented, fd=5 (on a pipe for the pipe). DescriptorControl.hpp or DescriptorControl.cpp alone from main does not build, since Udp.cpp then calls an undeclared or undefined function. The 0x20046601 and 0x80006601 checks on the socket have no negative control, since they pass on main too: run alone in the same scratch copy with all three files from main, they pass.
• A probe linked against the built libkernel.prx, with standard input from /dev/null (a character device): on main, ioctl(1, 0x5413, buffer) with standard output redirected to a file throws ioctl: request 0x00005413 on a file is not implemented, fd=1; 0x5413, 0x5401 and 0 throw on a file, a pipe and the character device, while 0x40007468, 0x20087468 and 0x00087468 (group t) fail there with ENOTTY; descriptor 4000 fails with EBADF, and a UDP socket with a null argument fails with EFAULT. With this change all of these, _ioctl, and 0x5413 with the upper 32 bits set fail with ENOTTY, while 0x20046601 (void, length 4) and 0x80006601 (in, length 0) still throw on a file, as FIOASYNC with a buffer does, and on a socket with a null argument they and FIONREAD fail with EFAULT, as on main.
• tools/select_tests.py picked 1349 of 1638 tests for this diff: 1334 passed, 0 failed, 15 skipped (the 13 video_out_flip_* tests and video_out_frame_dump_queue report no display or Vulkan device, as SDL's offscreen driver has no Vulkan here; agc_driver_stencil_ref_export reports that the device lacks VK_EXT_shader_stencil_export). guest_udp, guest_sendfile and guest_sce_net are among the passed.
• x86_64-w64-mingw32-g++ 16.2.0 -fsyntax-only -Wall -Wextra on the three changed .cpp files: no errors or warnings, as for their main versions. This is a syntax check, not a Windows build or run.
• tools/check_compile.py --base upstream/main: 3 translation units compile.
• tools/check_conventions.py --base upstream/main with this title and body: 0 errors, 1 warning (debt-length on the ioctl entry, 1649 characters; it is 1297 on main, already over the 1000 the warning allows).
• tools/pr_overlap.py --branch lists fix(libkernel): support socket fcntl duplication #7437 (socket fcntl duplication), which changes neighbouring lines of GuestKernelFcntl.cpp and the next debt entry; it implements other commands, so whichever of the two is merged second needs only a textual rebase. It could not compare the open PRs that conflict with main; merged with this branch, feat(aot): consolidate native ARM64 compiler, runtime and consumers #5021, feat: run PS4 games on the AGC driver — Limbo (CUSA01664) plays at 60 fps #6977 and Macos port #7201 conflict in the same files as they do with main, and feat: macOS on Apple Silicon host support (Rosetta 2 and MoltenVK) #7332 merges cleanly.
• Not run on Windows, macOS or a console.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: OpenOrbis PS4 toolchain sample net_http, Relinker rejects PS4 executables: SCE dynamic tables and data in the code segment (two ready branches) + remaining PS4 work #3093
• GPUs tested: n/a
• AI-assisted: yes

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 7437	core/libs/tests/GuestKernelFcntl.cpp L115-116 docs/dev/debt/libkernel.md L199-201

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 5021	core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 6044	core/libs/tests/GuestKernelFcntl.cpp
PR 6149	docs/dev/debt/libkernel.md
PR 6446	docs/dev/debt/libkernel.md
PR 6469	docs/dev/debt/libkernel.md
PR 6519	docs/dev/debt/libkernel.md
PR 6740	docs/dev/debt/libkernel.md
PR 6977	core/libs/prx/libkernel/Socket/src/Udp.cpp core/libs/tests/GuestKernelFcntl.cpp docs/dev/debt/libkernel.md
PR 7049	docs/dev/debt/libkernel.md
PR 7088	core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md
PR 7105	docs/dev/debt/libkernel.md
PR 7121	core/libs/prx/libkernel/File/include/DescriptorControl.hpp core/libs/prx/libkernel/File/src/DescriptorControl.cpp docs/dev/debt/libkernel.md
PR 7149	docs/dev/debt/libkernel.md
PR 7201	core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7332	core/libs/prx/libkernel/Socket/src/Udp.cpp core/libs/tests/GuestKernelFcntl.cpp docs/dev/debt/libkernel.md
PR 7509	docs/dev/debt/libkernel.md
PR 7554	docs/dev/debt/libkernel.md
PR 7556	docs/dev/debt/libkernel.md
PR 7605	core/libs/prx/libkernel/Socket/src/Udp.cpp docs/dev/debt/libkernel.md

[MarcZX10]

Manual overlap check against #7437:

• The implementation changes are separate: #7437 adds socket fcntl duplication commands, while this PR rejects malformed ioctl requests. In Udp.cpp, they modify the fcntl_nid_postfix and ioctl_nid_postfix paths respectively.
• Both PRs add assertions in GuestKernelFcntl.cpp's socket coverage and update adjacent entries in docs/dev/debt/libkernel.md, but they cover different behavior. I found no duplicate implementation or dependency. When rebasing/merging, retain both test additions and both debt updates.