Windows: crash reports are lost or land in a title's own file after the guest moves descriptor 2 (dup2, freopen); four catalog titles abort with no message; fix branch ready #4631

On Windows, once a title moves descriptor 2 with dup2(fd, 2) or freopen(..., stderr), the crash report and the terminate message of the process are lost, or are written into one of the title's own files. Four catalog titles abort this way on main with nothing on stderr, so their real stop cannot be read from a log. #4536 names this for SwanStationPS5 ("aborts ... with no report ... This PR does not change that"); this is a fix for that part.

What happens. CrashReport.cpp, the exception runtime (Runtime.cpp) and the [notification] writer in Notification.cpp write through GetStdHandle(STD_ERROR_HANDLE). The C runtime's _dup2 and _wfreopen close the handle behind descriptor 2, and in a relinked executable the Win32 standard handle keeps the closed value. Watched with gdb on main d592d54, with a breakpoint on NtClose for the value GetStdHandle(STD_ERROR_HANDLE) had at start:

• SwanStationPS5 (PPSA98510): closed from dup2_nid_postfix → _dup2. At the abort GetStdHandle(STD_ERROR_HANDLE) still returns that value and GetFileType of it is 0, so the report goes nowhere.
• ProsperoTV (PPSA99003): closed from freopen_nid_postfix → _wfreopen. By the abort the same handle value belongs to another file, and the whole report (terminate lines, FATAL: abort(), the stacks of all threads) is in the title's download0/profiles/10000000/logs/app.log.

In a test executable, which has the C runtime start-up, the same _dup2 instead moves STD_ERROR_HANDLE to the guest's new handle, the pipe. I did not look into why the two differ; the fix covers both.

Titles, relinked once, run with the libraries of main d592d54 and of the branch:

The poll stop is what #3644 is about. The ProsperoRadio stop was not visible in any log before. SwanStationPS5 and ProsperoRadio abort at a time that varies: in one 35 s run each was still running with both sets of libraries, and SwanStationPS5 once ran 45 s with main's.

Fix, ready to cherry-pick. DavidSamir:fix/libkernel-keep-host-stderr-handle-windows (7f11881, one commit on main d592d54, five files, +63/-1). I can't open it as a PR while #4532 is open (one open PR without a merged one).

Windows only. Before _dup2 replaces descriptor 0, 1 or 2, and before _wfreopen reopens a stream on one of them, HostStandardHandle_nid_no_patch takes the Win32 standard handle of that descriptor: a duplicate when it is still the handle the descriptor owns, the handle itself when it was already kept. After the call KeepStandardHandle_nid_no_patch sets it as the standard handle again. The host's reports keep going where the process's standard error went at start, the guest's descriptor goes where the guest moved it, and the two no longer share a handle that the other side can close. A process started without standard handles is left alone. Later moves of the same descriptor reuse the kept handle, so nothing accumulates.

This is the same direction as #4536, which keeps sceKernelDebugOutText on the host's standard error after the guest's dup2; it does not depend on it. The two touch the same dup2 lines in Stdio.cpp and both add a dup2_nid_postfix declaration to GuestFilesystem.cpp, so whichever lands second needs a small rebase, which I can do.

Not covered: a guest that closes descriptor 2 and opens another file onto it, and fclose(stderr). Linux is not changed.

guest_filesystem gains a Windows check for both paths: with the host's descriptor 2 on a temporary file, the guest moves descriptor 2 onto a pipe with dup2, or onto a file with freopen; a write through GetStdHandle(STD_ERROR_HANDLE) must reach the temporary file, the standard handle must not be the guest's new handle, and a guest write(2, ...) must reach the pipe or the file.

Checked on Windows (7f11881 on main d592d54, Windows 11 10.0.26300, WinLibs GCC 15.2.0 r7, Release):

• With the four source files as on main and the new check kept, guest_filesystem fails at it (Filesystem check failed at line 231). With the fix it passes 5/5 (--repeat until-fail:5); guest_standard_streams, file_streams, guest_dup, guest_fiopen, guest_poll and windows_exceptions pass.
• The four titles above. With the branch, ProsperoTV's download0 holds no file with a crash report in it.
• Ten more catalog titles, 35 s each with both sets of libraries: ccleste (PPSA59322), RommPS (PPSA76677), PSXS5 (PPSA97510), DOOM (PPSA99666), OpenRCT2 (PPSA99728), ScreenTester5 (PPSA69420), Prospero Vibrate (PPSA99920), Yamagi Quake II (PPSA99007), Middle Ages Ping Pong (PPSA28319) and SymphonyStation5 (PPSA17641) end the same way in both, with the same number of stderr and stdout lines. PSXS5 reports its poll stop in both.
• Full ctest --parallel 4: 971 tests. 109 failed in my first run, in a shell that still had this machine's PYTHONHOME set, which breaks the Python-driven tests; rerun without it, all 109 pass. One of the 109 is not Python-driven: guest_audio_in_capture exited 0xc0000409 once in the parallel run and passed 6 of 6 afterwards; I did not find the cause.
• tools/check_conventions.py --base origin/main passes. tools/pr_overlap.py --branch lists fix(libkernel): share the directory state between duplicated descriptors on Windows #3872, fix(libkernel): give guest sockets ordinary descriptor numbers below FD_SETSIZE #4391, fix(libs): share Windows flock state between duplicated descriptors #4516 and fix(libkernel): keep sceKernelDebugOutText off a redirected guest stderr #4536 for neighbouring lines in Stdio.cpp and FileStream.hpp; none of them changes the standard handles.

Checked on Linux (same commit, WSL2 Ubuntu 24.04, GCC 13.3.0, Release): guest_filesystem and guest_standard_streams pass 3/3. Only those targets and libs were built there; the new check is Windows only.

Not checked on a console, with a --windows-gui build, or with a commercial title.

AI-assisted: yes