feat(tools): run titles repeatably and report their outcome

[LeandroLP23]

What

A tool for running titles you already have the same way every time, so a run can be attached to a PR or an issue and compared with an earlier one.

python tools/title_run.py --games <directory> [ID ...] finds extracted packages (eboot.bin and sce_sys/param.json) under --games, unwraps the SELFs with tools/unwrap_self.py, relinks with the title's flags and lays the title out as docs/user/USAGE.md describes. It then runs each title --runs times (default 2: a cold run with empty AnyPS5 shader cache and pipeline journal, then warm runs) and decides each run's outcome by itself:

Outcome	When
relink-failed	the relinker (or unwrapping) fails
crashed	the title ends before the deadline with a non-zero code
exited	it ends before the deadline with code 0
stalled	alive, but no aps5-summary.jsonl line in the last 20 s (no presents), or too few presents for a frame to be dumped
blank	alive and presenting, but the last dumped frame is one colour
rendering	alive, presenting, and the last frame has content

It writes summary.json and a summary.md table with fixed columns (title, ID, version, cold, warm, FPS, first error) under build/title-runs/<commit>-<time>/, keeping the first, middle and last dumped frames of each run. --compare <summary.json> adds a Before column and lists regressions, improvements and titles missing from the new run; when IDs are given, only those are compared.

• Per-title settings live in tools/titles.json: extra relink flags, fixed_base (mapped to --windows-fixed-base/--linux-fixed-base), executables for LoadExec targets, run length and environment. It holds the flags the catalog titles need today (VLC, Ruffle Flash and XPSemu at 0x400000, RetroArch's picker/picker.bin, the --trap-sdk-syscall-stubs titles). .gitignore gets !tools/titles.json, next to the existing !demo/web/i18n/*.json.
• Processes. On Windows a Job Object follows the title and every process it starts (sceSystemServiceLoadExec hand-overs included), and the run's exit code is that of the newest process whose image is inside the title's folder. On Linux it is a process group. A title is always ended at the deadline (close, 15 s grace, then terminate), and nothing is left running.
• Not in the way. Audio is muted (SDL_AUDIODRIVER=dummy). On Windows there is no console window, the title's windows never get focus and go behind every other window as soon as they are created (a WinEvent hook, so they do not flash in front first). They keep their position and can be brought forward from the taskbar. They are not minimized, because presents stop while a window is minimized.
• Caches. Mesa and NVIDIA shader caches are pointed into the run's folder. AMD's Windows VkCache cannot be redirected and is left alone; summary.json records it as not isolated.
• Never sources data. It only reads packages the user points it at.

Unit tests: tools/tests/test_title_run.py, registered as title_run. They cover outcome classification from synthetic logs, summaries and BMPs (including truncated ones), relink arguments, the report and comparison, package discovery, and the process session with real child processes. Docs: a section in docs/dev/TRACING.md.

Tested

Windows 11 (26300), Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), Ryzen AI 7 350, WinLibs GCC 15.2.0, Release.

• tools/tests/test_title_run.py on Linux and macOS (GitHub ubuntu-latest and macos-latest, Python 3.12, on a throwaway fork branch, because this pull request's CI skips the build jobs): 10 runs of 10 on each. The first try failed on macOS, where killpg on a group whose processes are all exiting fails with EPERM; alive(), close() and kill() now treat that as still exiting.

• python tools/tests/test_title_run.py: 24 OK. ctest -R '^(title_run|test_registration|conventions_doc_links)$': 3/3. tools/check_conventions.py --base origin/main passes.

• The local homebrew catalog, 66 titles, main 5859889b2 with this branch, one 60 s run each, then the 22 titles that did not reach rendering again with --runs 2 --compare <first summary.json>. Nothing was left running after either batch. Windows Firewall asked about titles that open a port (answering Cancel does not affect the run); titles are now prepared at a fixed path so it asks once per title.

• The 22 titles again on main 1c523bdb4, --runs 2 --compare against the table below: RommPS now renders cold, Mortalfin's cold run hit the 10 s GPU watchdog (has not reached submission 10) like ProsperoAI and RetroArch do, everything else matched. ProsperoPuzzles renders with fix(agc): run the unwatch hook when a watched import leaves the write watch #6308 on top (29.2 FPS, cold and warm). ScreenTester5 had shown rendering from two presents just before the deadline, which led to the frame rule for stalled.

  • rendering: 48. 40 on the first pass. SuperTuxKart (PPSA69421) and ProsperoEden (PPSA99008) needed --trap-sdk-syscall-stubs, now in titles.json. Mortalfin (PPSA99141) and Nuvio (PPSA99997) render on both runs of the second pass. ProsperoAI (PPSA99004), RetroArch (PPSA99169), RommPS (PPSA76677) and PS5 OpenGL Showcase (PPSA99005) render only warm: cold, they stop on the PS5 RADV port's 10 s submission wait while their first pipelines build, which the Cold and Warm columns now show directly.
  • The second pass, as the tool wrote it:

Title	ID	Version	Cold	Warm	FPS	First error	Before
XashPS5 V1.3	PPSA19111	01.030.000	exited	exited	0.0		exited
Middle Ages Ping Pong	PPSA28319	01.000.000	crashed	crashed	0.0		crashed
ScreenTester5	PPSA69420	01.000.000	stalled	stalled	0.0		stalled
SuperTuxKart	PPSA69421	00.001.000	rendering	rendering	6.1		relink-failed
RommPS	PPSA76677	01.001.001	exited	rendering	8.7	[debug:0] [RommPS] radv/ps5: the GPU has not reached submission 10 (at 9) in 10 s	exited
ProsperoAI	PPSA99004	01.005.000	crashed	rendering	7.3	radv/ps5: the GPU has not reached submission 2 (at 1) in 10 s	exited
PS5 OpenGL Showcase	PPSA99005	01.000.001	crashed	rendering	4.0		exited
ProsperoPuzzles	PPSA99006	01.000.010	crashed	crashed	0.0	what(): AGC graphics: storing storage image results pending over memory outside the write watch without the guest bytes	exited
ProsperoEden	PPSA99008	01.000.110	rendering	rendering	59.7		relink-failed
vkQuake	PPSA99010	01.000.000	exited	exited	0.0		exited
PS5 FSR4 Showcase	PPSA99011	01.000.010	exited	exited	0.0		relink-failed
Homebrew UI Lab	PPSA99050	01.000.010	crashed	crashed	0.0		exited
Surround Sound Studio	PPSA99051	01.000.003	crashed	crashed	0.0		exited
OpenNOW PS5 - Prototype	PPSA99082	00.002.038	stalled	stalled	0.0		stalled
Mortalfin	PPSA99141	01.000.002	rendering	rendering	81.5		exited
RetroArch	PPSA99169	01.000.000	exited	rendering	38.8	[debug:0] [Frontend Picker] radv/ps5: the GPU has not reached submission 12 (at 11) in 10 s	exited
PS5VR	PPSA99177	01.002.000	stalled	stalled	0.0		stalled
Quake III Arena	PPSA99303	01.001.000	exited	exited	0.0		exited
Wind Waker	PPSA99309	01.000.001	exited	exited	0.0		exited
Porpoise (DolphinPS5)	PPSA99764	03.000.000	stalled	stalled	0.0		stalled
PICO-8	PPSA99808	01.001.002	crashed	crashed	0.0		crashed
Nuvio	PPSA99997	01.000.007	rendering	rendering	59.9		stalled

• Before is the first pass. Its exited for the crashing titles was a bug found by that pass: a crash brings Windows Error Reporting into the job, and its exit code 0 was taken for the title's. The exit code now comes only from processes inside the title's folder, so these show as crashed (for example ProsperoPuzzles with 0xC0000409, the abort of AGC: since #6060, ProsperoPuzzles (PPSA99006) stops at 5 s with the storage-image write-back throw on Windows #6301).
• XashPS5, Wind Waker and Quake III Arena exit by design without the user's own game data (the valve folder, a GZLE01 disc image, .pk3 files); PICO-8 needs elfldr; FSR4 Showcase gives up on the build time of its FSR4 pipelines in a first run (AGC: PS5 FSR4 Showcase (PPSA99011) gives up in its first FSR4 pass while the queue worker builds 130k-800k-word compute pipelines (up to 66 s each on AMD) #6118). OpenNOW reached its sign-in screen at 25.8 FPS on d8888df9b (catalog) and stops after a few presents here: its sign-in screen redraws only when its view changes and still answers input, so stalled is expected there (OpenNOW PS5 (PPSA99082) presents only on view changes; older builds' continuous presents were likely padding residue #7501).
• A package whose eboot.bin is neither an ELF nor a SELF gives relink-failed with Refused: file is smaller than the SELF header area; the batch carries on and writes summary.json.

Not tested: real titles on Linux and macOS, NVIDIA, Intel. Known limits: a LoadExec child that starts and exits within one poll (0.1-0.5 s) is not seen, and the parent's exit code is used; app0 is hard-linked to the user's package when the file system allows it; two invocations sharing one --out share work/<ID>.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none (a test tool; the example runs cover VLC PPSA85300, ProsperoLight PPSA99002, ProsperoPuzzles PPSA99006 and RetroArch PPSA99169)
• GPUs tested: AMD Radeon 860M (RDNA 3.5), driver 32.0.13058.8001, Windows 11
• AI-assisted: yes (Claude Code)

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 674	CMakeLists.txt
PR 1835	CMakeLists.txt
PR 5021	.gitignore CMakeLists.txt
PR 6044	CMakeLists.txt
PR 6746	CMakeLists.txt
PR 6879	CMakeLists.txt
PR 6946	CMakeLists.txt
PR 6947	CMakeLists.txt
PR 7039	CMakeLists.txt
PR 7201	CMakeLists.txt
PR 7332	CMakeLists.txt
PR 7417	docs/dev/TRACING.md
PR 7604	docs/dev/TRACING.md
PR 7642	CMakeLists.txt

[LeandroLP23]

Two questions before this gets reviewed, so I can adjust it first:

1. @oneandonlydean @adriadam10: is tools/titles.json fine in the repository? It holds only title IDs, relink flags (--trap-sdk-syscall-stubs, --windows-fixed-base, --executable for LoadExec targets), run length and environment for the catalog homebrew, and no game files. Without it, everyone running the catalog has to rediscover those flags. If you'd rather keep per-title data out of the tree, --titles already takes any path, so it can become a local file and the tool stays.
2. @AlxcBTW: summary.json is meant to be importable by the blocker map. Per title it has name, version, seconds and runs[]. Each run has run, cache (cold/warm), outcome, exit_code, seconds, presents, fps, first_error, warnings and the kept frames. The top level adds build (commit, dirty, note), machine (os, gpu, driver, cpu), runs, seconds and driver_cache. Is anything missing that the map would need, or named in a way that does not fit it?

AI-assisted