fix(libSceVideoOut): return VIDEO_OUT_ERROR_INVALID_VALUE for a flip rate outside 0..2

[xXPBSXx]

What

PPSA13356 (Raiden III) calls sceVideoOutSetFlipRate(handle, 60) at start-up. sceVideoOutSetFlipRate accepted only 0, 1 and 2 and threw for any other rate, so the title aborted:

what():  sceVideoOutSetFlipRate: VIDEO_OUT_ERROR_INVALID_VALUE

(PPSA13356_rU/run_081948.err lines 8-9, main + #5038, without this change; the rate 60 was read with a local log line in PPSA13356_rR/run_080252.err line 6.)

The function now returns VIDEO_OUT_ERROR_INVALID_VALUE for a rate outside 0..2 and leaves the stored rate unchanged.

• Title evidence: PPSA13356, the abort above. With this change (branch head 76e5edcac) + libScePad: replace NotImplemented stubs and fix return-0 consistency #5038 the run no longer stops at sceVideoOutSetFlipRate; it goes on and ends at a different place, what(): NGS2: invalid voice handle (PPSA13356_rV/run_083704.err lines 29-30, and in two repeats run_083940.err, run_084444.err). That stop is not part of this pull request.
• Not known: whether the console accepts 60 (and treats it as something other than the error) or returns VIDEO_OUT_ERROR_INVALID_VALUE for it. There is no source for it in the repository. The change relies on the title tolerating the error; docs/dev/debt/libSceVideoOut.md has an entry that says so.
• scePadUnknown_n3kSX62fgNo (libScePad: replace NotImplemented stubs and fix return-0 consistency #5038) has to be merged for PPSA13356 to reach this call; this change does not depend on it in code.

Tested

Windows 10 (19045), MinGW-w64 GCC 15.2.0, Release. Branch rebased onto main 1e7272196 (two commits, no merge):

• video_out_flip_tests controls (changed: calls sceVideoOutSetFlipRate with 3, 60 and -1, expects VIDEO_OUT_ERROR_INVALID_VALUE and an unchanged stored rate) passes with this change; the other four flip tests (lifetime, reopen, overlay, unopened) pass too. They ran with SDL_VIDEODRIVER=windows on an RTX 4080 SUPER, because under ctest the SDL driver is offscreen and the tests are skipped.
• Without the change to Output.cpp (taken from main, test kept) controls fails: sceVideoOutSetFlipRate: VIDEO_OUT_ERROR_INVALID_VALUE, exit code 1.
• python tools/check_conventions.py --base upstream/main with this title and body passes.
• Title run PPSA13356, test conditions: tree = branch head 76e5edcac + libScePad: replace NotImplemented stubs and fix return-0 consistency #5038 (merge 6a4d76b8e); libraries ninja -C build -j4 relinker plus all 126 core/libs/libs/*.prx of that tree, finished 08:36:53 local time (libSceAgcDriver.prx 08:36:23); relinker.exe 2026-10-14 07:35:29; app.exe relinked 08:37:00 with relink_title_trace.ps1 -Repo K:\AnyPS5\wt-sockets (relinker.exe --windows --registry --trace-imports, no -ModuleDir); run run_title.ps1 -Seconds 90. Result: past SetFlipRate, then NGS2: invalid voice handle as above.
• Not tested: Linux (one source for both hosts, only Windows was built), a console, GPUs other than the RTX 4080 SUPER.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt (docs/dev/debt/libSceVideoOut.md)
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source (none added)
• Generic behaviour, not specific to one title
• Depends on: none (PPSA13356 also needs libScePad: replace NotImplemented stubs and fix return-0 consistency #5038 to reach this call)
• Unblocks: PPSA13356 (Raiden III) at sceVideoOutSetFlipRate; no bring-up discussion known to me
• GPUs tested: NVIDIA GeForce RTX 4080 SUPER, Windows 10 (19045); driver version not recorded
• AI-assisted: yes

AI-assisted (Claude Code).

🤖 Generated with Claude Code

[xXPBSXx]

@adriadam10 a single, short request for review of our five open pull requests (all CLEAN, mergeable, Conventions/Eligibility/Intake checks green on each; each description names its title, the log line and the test that fails without the change):

• fix(libSceVideoOut): return VIDEO_OUT_ERROR_INVALID_VALUE for a flip rate outside 0..2 #5691 sceVideoOutSetFlipRate, Raiden III x MIKADO MANIAX (PPSA13356); the title reaches this call only with libScePad: replace NotImplemented stubs and fix return-0 consistency #5038 merged, this change does not depend on it in code
• fix(libSceAgcDriver): return 0 from sceAgcDriverRegisterDefaultOwner #5834 sceAgcDriverRegisterDefaultOwner, Broken Sword - Shadow of the Templars: Reforged (PPSA21480)
• fix(libSceAgcDriver): zero the overhanging record of a vertex fetch #5926 vertex fetch overhang in CopyZeroPaddedDrawInput, The Mound: Omen of Cthulhu (PPSA26223)
• fix(libkernel): make close wait for a positioned transfer on the same descriptor #5938 close against a positioned transfer, Cat Quest III (PPSA15065)
• fix(libc): name a guest abort() in the crash report #5957 crash report names a guest abort() (diagnostic only), Sifu (PPSA03001)

Thank you for the reviews and merges on #4778, #5020 and #5143. No reply is needed beyond whatever review you see fit; I will not post a second request.

AI-assisted (Claude Code).

[adriadam10]

Checked: the full diff, video_out_flip_tests (controls, lifetime, reopen, overlay, unopened) on Linux with a display, and the changed controls test fails without the Output.cpp change. KytyPS5 and prosper also return VIDEO_OUT_ERROR_INVALID_VALUE for a rate outside 0..2, and the constant matches. The unverified rate 60 is recorded in docs/dev/debt/libSceVideoOut.md. Merging once the full Build is green.

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
#3891	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4040	core/libs/prx/libSceVideoOut/src/Output.cpp docs/dev/debt/libSceVideoOut.md
#4156	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4396	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4397	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4422	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4423	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4424	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4520	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4521	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4522	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4523	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4566	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4599	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4718	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4740	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4749	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4760	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#4813	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#5021	core/libs/prx/libSceVideoOut/tests/Flip.cpp
#5877	docs/dev/debt/libSceVideoOut.md
#6044	core/libs/prx/libSceVideoOut/tests/Flip.cpp