fix(libc): translate host errno from fopen, freopen and fopen_s

[AlexOfTheForest]

What

libc's stream-opening calls passed the host's errno to the title untranslated. On Linux a symlink loop reached the title as 40 and an over-long name as 36, which are EMSGSIZE and EINPROGRESS on FreeBSD 11.4 instead of ELOOP (62) and ENAMETOOLONG (63). fopen_s returned ENOENT for every failure. opendir, realpath, chdir, chmod, utimes, lutimes and chflags already return the FreeBSD values; only the fopen family leaked.

• fix(libc): GuestErrnoFromHost_nid_no_patch (libc/src/HostErrno.cpp) maps a host errno to FreeBSD's:

  • 1–34 are shared, except EAGAIN, which becomes 35;
  • ELOOP, ENAMETOOLONG, ENOTEMPTY, EDQUOT, EOVERFLOW and EDEADLK get their FreeBSD numbers;
  • anything else is EIO.

  fopen uses it. On Windows OpenNative already sets host values from WindowsOpenError_nid_no_patch, so they pass through the same table.

• fix(libc): freopen uses the same translation for its failed reopen.

• fix(libc): fopen_s returns the errno fopen set. C11 K.3.5.2.1 only requires a nonzero value; Microsoft's CRT, built on Dinkumware like the console's libc, returns errno. Recorded in docs/dev/debt/libc.md as unverified on console.

Tested

Linux x86-64, GCC.

• New guest_fopen_errors test (UNIX only, since it creates symlinks):
  • fopen, freopen and fopen_s with a self loop, a path through a loop, a 300-byte name and a path under it;
  • a missing file, a directory opened for writing and an invalid mode;
  • opens that still succeed.
• Reverting each fix makes its part of the test fail.
• Existing tests pass: file_streams, file_position, guest_fiopen (including its fopen_s checks), access, directories, fdopendir, standard streams, working directory, host streams and statvfs.
• No function goes back to being a stub in the progress report.
• Not tested on Windows or macOS.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6975	docs/dev/debt/libc.md L85
PR 7081	core/libs/prx/libc/src/FileAndHeap.cpp L320-321

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 5021	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 5303	docs/dev/debt/libc.md
PR 5668	core/libs/prx/libc/CMakeLists.txt core/libs/prx/libc/src/FileAndHeap.cpp
PR 6044	core/libs/prx/libc/CMakeLists.txt
PR 6061	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 6222	core/libs/prx/libc/CMakeLists.txt docs/dev/debt/libc.md
PR 6287	core/libs/prx/libc/src/FileAndHeap.cpp
PR 6565	docs/dev/debt/libc.md

[screamm]

Windows run of head 5433b32b7, merged onto main c9e6c45de (clean; also stacked with #7070).

• Host: Windows 11 Pro 26220, Ryzen 7 5700X, RTX 4070 (driver 596.21), WinLibs MinGW-w64 GCC 15.2.0 r7, Ninja, Release.
• Build: the full build and libs pass, no warnings in HostErrno.cpp or FileAndHeap.cpp.
• guest_fopen_errors is registered under if(UNIX), so it is not built on Windows and I could not run it. guest_fiopen and file_streams pass 3/3.
• Errno table: I compiled GuestErrnoFromHost_nid_no_patch with MinGW and ran every constant through it. Against FreeBSD 11.0 sys/errno.h: EAGAIN 11 -> 35, EDEADLK 36 -> 11, ENAMETOOLONG 38 -> 63, ENOTEMPTY 41 -> 66, ELOOP 114 -> 62, EOVERFLOW 132 -> 84. The values 1-34 pass through (ENOENT 2, EACCES 13, EEXIST 17, ENOTDIR 20, EISDIR 21, EINVAL 22, EMFILE 24, ENOSPC 28). WindowsOpenError_nid_no_patch returns only ENOENT, EACCES, EEXIST, EMFILE, ENOMEM, ENOSPC and EINVAL, so the EIO fallback is never taken by a Windows open.
• Probe: a throwaway Windows test (not in the PR) with 15 cases x fopen/freopen/fopen_s. With the PR, fopen and freopen return the same errno as main in all 15 cases. fopen_s differs in 6: main returns 2 for every failure, the PR returns 13 (directory opened for writing or reading, read-only file with w and r+), 17 (wx on an existing file) and 22 (invalid mode). With FileAndHeap.cpp reverted to main the fopen_s column goes back to 2.
• On Windows a few cases still differ from FreeBSD, on main as well: a 300-character name gives 2 (FreeBSD 63), a file used as a directory gives 2 (20), and a directory opened for writing gives 13 (21). They come from WindowsOpenError_nid_no_patch (ERROR_FILENAME_EXCED_RANGE and ERROR_PATH_NOT_FOUND both give ENOENT, ERROR_ACCESS_DENIED for a directory gives EACCES), so the ENAMETOOLONG and ELOOP fixes do not show on Windows. Not for this PR; mapping those Win32 errors would be a follow-up.
• Non-GPU suite (ctest -j8 --timeout 120 -E '^agc_driver_|^video_out_flip'): 884 tests (my throwaway probe included), 872 pass, 2 skipped, 10 failed on the first pass: the conventions_function_bodies scan and its 5 shards, hardware_oracle_compare, guest_none_relocations and tls_function_coverage_register_address_loads timed out under load, and guest_sce_net aborted once at -j8. All pass on serial reruns (shard 4 on its second serial try, 20.5 s against its 30 s limit; guest_sce_net 5/5). None of them touches code from these PRs
• tools/check_conventions.py --base c9e6c45de and git diff --check pass. There are no new exports.

Titles. fopen is imported by 50 of my local relinked titles and freopen by 28. I ran four for 60 s each on main and on this PR with a local diagnostic libc that logs each failed fopen/freopen/fopen_s and the errno returned:

Title	failed fopen (main / PR)	errno returned (main / PR)
PPSA99203	6 / 6 (/data/PCSX2/boot.log, pf.log, pid.txt for writing; en.txt, gs.ini, usb-games.txt)	2 / 2
PPSA99169 RetroArch picker	4 / 4 (/app0/trace.txt, picker/test-run.txt, config/frontend.cfg, picker/picker-test.txt)	2 / 2
PPSA01153	1 / 1 (/download0/etahen_jailbreak, write)	2 / 2
PPSA68091 Ruffle	0 / 0	none

No freopen or fopen_s call in any of them. Every title got 2 (ENOENT) on both sides, which is also FreeBSD's value for these, so the codes this PR fixes did not occur. All runs were killed at the limit with no stop line.

Review: no comments, no new exports or NIDs, no title-specific code. Since the symlink and long-name cases can only fail on a Linux host, a variant of the test for Windows (missing file, directory opened for writing, invalid mode, fopen_s) would give the change some coverage here.

Not checked: Linux, macOS, PS5 hardware, freopen with a null file name, any title that hits ELOOP/ENAMETOOLONG (none local).

Written with Claude Code assistance (validation by a subagent, reviewed in this session).