fix(libkernel): keep sceKernelDebugOutText off a redirected guest stderr

[orishamir]

What

This carries the second commit of #1870 by @TheAggie, with its author kept, rebased onto current main, plus one small commit of mine. #1870 was closed on Oct 9 because ccleste did not reach dup2 on main yet; #2478 then added dup and dup2 without this commit.

Titles built on the PS5 Vulkan template call ps5_klog_capture_stderr (mihawk-99/PS5_PayloadSDK platform/src/klog.c). It creates a pipe, starts a thread that passes every line read from the pipe to sceKernelDebugOutText, and moves standard error onto the pipe with dup2(pipe, 2). Guest descriptors are host CRT descriptors, so the dup2 also moves the host's descriptor 2, the one sceKernelDebugOutText writes to. The reader thread then reads its own output back, each round adds a prefix, and once the pipe is full the reader blocks writing into the pipe that only it reads. From then on every host thread that writes to stderr waits for that stream's lock.

On main 0ef9dc9, ccleste (PPSA59322) and RommPS (PPSA76677) print no error. Their last line is Vulkan device ready, and no swapchain is created. The thread stacks on Windows, taken with gdb at 35 s, are the same for both titles:

• The reader thread is in sceKernelDebugOutText → fprintf → fputc → _fflush_nolock → WriteFile, which does not return.
• The thread creating the Vulkan device waits for the stderr lock in fprintf, called from Graphics::PrepareImportWatch. That call sits between Vulkan device ready and Surface capabilities in the VulkanDevice constructor.

Changes:

• feat(libkernel): implement dup2 #1870's commit. Before the first guest dup2 onto descriptor 2, the host's descriptor 2 is duplicated and kept (File::HostStandardError). sceKernelDebugOutText writes to that copy from then on. The guest's own writes to descriptor 2 still reach its pipe.
• Mine. On Windows, the _dup(2) goes through WithoutParameterHandler, like the other descriptor calls in Stdio.cpp. A GUI build started without standard handles then gets the commit's dup2: cannot keep the host standard error error instead of the invalid-parameter abort.
• Test (guest_filesystem, from feat(libkernel): implement dup2 #1870). The host's descriptor 2 is on a temporary file and a guest pipe is moved onto descriptor 2. sceKernelDebugOutText output must reach the file, and a guest write(2, ...) must reach the pipe.
• Rebase. dup2 was rewritten by feat(libkernel): implement dup and dup2 #2478. In both branches the call that keeps the descriptor now comes right before the host _dup2/dup2; on Windows it comes after RejectDirectoryDuplicate. The test needed a dup2_nid_postfix declaration in GuestFilesystem.cpp.

Not changed: what dup2 onto descriptor 2 returns. The SDK says the console refuses it ("A title may not dup2 (the console refuses it with EPERM)") and then moves only the stderr stream.

Tested

Windows 11 (26300), WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON, RTX 4080 (driver 616.56), on main 0ef9dc9:

• guest_filesystem and guest_dup pass 3 of 3. Revert-pin: with Stdio.cpp, File.hpp and Notification.cpp as on main, guest_filesystem fails at the pipe read (Filesystem check failed at line 245), because the pipe also receives [debug:1] kept.
• Full ctest: 966 of 966 pass, and check_skips passes.
• ccleste and RommPS, with a frame saved every 30 presents:
  • On main, neither saves a frame in 90 s.
  • With this PR, both log Swapchain created and save the 30 frames by present 870, 14.5 s after the first, at about 60 presents a second. ccleste shows its title screen. RommPS shows its own "RomM Sync isn't running" screen, because it needs a HEN payload loader on port 9021.
• Whole homebrew catalog: the 58 titles that download and unpack today, 45 s each with --trap-sdk-syscall-stubs, main 0ef9dc9 against main + this PR.
  • ccleste and RommPS save no frame on main and 4 with this PR (the sweep saves at most 4).
  • Quake3e (PPSA99303) needs the player's baseq3 data. On main it ran 45 s with nothing on stderr; on main 3d8ffeb it had exited at once, so the hang depends on timing. With this PR its log reaches stderr and it exits with code 0 at once.
  • Every other title stops where it did. ProsperoRadio, ProsperoEden and ProsperoLichess print a different last line from run to run on main itself (checked with a second main run), and none of them imports dup2.
  • SwanStationPS5 also moves descriptor 2. It aborts after 35-36 s in both arms with no report. The Windows crash report is written to STD_ERROR_HANDLE (CrashReport.cpp), which the CRT's _dup2 onto descriptor 2 also replaces, as feat(libkernel): implement dup2 #1870 noted. This PR does not change that.

The branch was then rebased onto main d592d54, where these files are unchanged apart from unrelated symbolic-link checks in GuestFilesystem.cpp. On the pushed head, on Windows, guest_filesystem and guest_dup pass 3 of 3, and ccleste creates its swapchain and saves 17 frames in 30 s.

Linux, in an Ubuntu 24.04 container (GCC 13.3, lavapipe, X11):

• On main, ccleste stops at the same Vulkan device ready, writes nothing to stderr, and does not exit on SIGTERM. gdb shows the reader thread in write() from sceKernelDebugOutText.
• With this PR it creates the swapchain and reports its next stop, AGC graphics: subgroup capability 61 is unsupported for shader stage 1, which is a lavapipe limit.

tools/check_conventions.py --base origin/main passes. tools/pr_overlap.py --branch lists #3872, which removes the RejectDirectoryDuplicate line just above the new call. Neither PR depends on the other.

@TheAggie, this is your commit. If you would rather resubmit it yourself, say so and I will close this one.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA59322, PPSA76677
• AI-assisted: yes (Claude Code Opus 5.5 xhigh)

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
#3872	core/libs/prx/libkernel/File/src/Stdio.cpp L548
#4640	core/libs/prx/libkernel/File/src/Stdio.cpp L548-549
#4672	core/libs/tests/GuestFilesystem.cpp L66

[xXPBSXx]

Windows test: head fc0d46ba0 merged onto current main 2a5eca11c (clean merge, no conflicts). Windows 10, MinGW GCC (WinLibs), Ninja Release, -j2: full build 0 errors; ctest with the filter guest_filesystem: guest_filesystem passed (1 test(s), 0 failed); python tools/check_conventions.py --base upstream/main passed. I did not run the new tests against main without the change.

AI-assisted (Claude Code).

[MarcZX10]

Coordination note: I carried David’s offered #4631 fix into #4640, preserving his commit author. It separately preserves Win32 GetStdHandle outputs across guest dup2/freopen so crash and termination reports survive; this PR keeps sceKernelDebugOutText off guest stderr. The changes are complementary, touch adjacent Stdio.cpp/GuestFilesystem.cpp code, and do not depend on each other. #4640 documents the integration point.