feat(tools): generate Export.cpp skeletons from missing NIDs

[Pouare514]

What

New contributor tool tools/stubgen.py: given a missing-NIDs list (one 11-char NID per line) plus a NID->name CSV mapping (NID name per line, as in aerolib.csv), it emits an Export.cpp skeleton of throwing stubs following repo conventions: a database name that recomputes to the NID (same SHA1+salt scheme as tools/nid_names.py) is emitted as the real name directly as a plain export, C/C++ runtime names (leading _ or bare CRT like wcstombs_s/swscanf_s) in <real>_nid_postfix plain-definition form, anything else (no DB hit or NID mismatch) as collision-free UnknownNN exported through APS5_EXPORT, every body a NotImplemented_nid_no_patch("NID") throw with a dummy return. It also emits the TechnicalDebt line per stub (unknown signature for real names, unknown name, signature for UnknownNN) with the target path/lib derived from --module (--td-path/--td-lib override, --td-out/--td-only for TD output). Real bodies are intentionally out of scope and stay hand-written. Stdlib only, no new dependencies, no comments in code or output.

Tested

• Ran on synthetic samples: verified sce* emits plain, _-leading and bare-CRT names (_Thrd_sleep, wcstombs_s, swscanf_s, __cxa_*, _ZTV*, pthread_*, malloc) emit _nid_postfix, unknown NIDs and mismatched DB names fall back to UnknownNN + APS5_EXPORT, TD lines use derived core/libs/prx/MODULE/Export.cpp paths (libc/libkernel mapped, overrides work), no // name lines in output.
• python syntax check of the committed file, git diff --check clean.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• AI-assisted: yes

[oneandonlydean]

Thanks. Generating the throwing stubs from the NID database is useful, but the hardcoded bodies would spread semantics that don't match Sony's CRT into every Export.cpp made with it:

1. Same as feat(libs): implement C runtime re-exports for libSceAmpr #545: localeconv returns the host struct (Sony's has decimal_point at offset 0x48), wchar_t is the host width while the guest's is 16 bits, and __udivti3 returns 0 for a zero divisor where the real one traps.
2. _Mtx_init ignores the type. Sony's makes the mutex recursive when type & 0x100, and _Mtx_lock maps EDEADLK to 3 and other errors to 4 instead of always returning 0 (from libc.prx, also discussed in feat(libc): implement the C11 mtx and cnd functions #492).
3. vsnprintf/vsscanf pass the guest va_list straight to the host, which is a different type on a Windows host.
4. The generated code has // name comments and the tool itself has many comments; CONVENTIONS allows none outside TechnicalDebt.
5. The description says the DB file is what tools/nid_names.py produces, but feat(tools): suggest real names for Unknown NID stubs from hash-verified DB #542 prints a table or JSON; it's the CSV that works as input.

I'd keep the generator to stubs named from the database and leave real bodies to hand-written PRs with tests. @boykopovar, would you want a generator like that in tools/ at all?

[Pouare514]

Reworked per review: the generator now emits throwing stubs only — the whole hardcoded CRT body table is gone, so no non-Sony semantics can spread. Database names become prefix+CamelCase throwing stubs, everything else collision-free UnknownNN. All comments and the module docstring are removed (shebang only), as is the generated // name line. The description now says the DB input is a plain NID-name CSV like aerolib.csv. Description also updated accordingly.

[oneandonlydean]

Thanks. The bodies and comments are gone, so the generator now only writes throwing stubs, and the description matches the input format (the same whitespace-separated file tools/nid_names.py reads).

One thing about the names it generates: a NID with a database name becomes a made-up prefix + CamelCase name exported through APS5_EXPORT (for example sceAmprLocaleconv). In #301, #338 and #407 the stubs ended up with the real name as a plain export instead, since the patcher recomputes the NID from it. So it would be more consistent if the tool wrote the real name directly, and the _nid_postfix form for C and C++ runtime names. On #407 that also turned out to matter on Linux, where a plain export named like a libstdc++ symbol interposes the host's own. It could also write the TechnicalDebt line for each stub ("unknown signature" or "unknown name, signature"), since CONTRIBUTING asks for one.

The open question is still @boykopovar's: whether a generator like this belongs in tools/ at all.

[Pouare514]

Reworked per review:

• Verified DB names are emitted as the real name directly as a plain export (no APS5_EXPORT), NID verified with the same SHA1+salt scheme as tools/nid_names.py.
• C/C++ runtime names (leading _, or bare CRT like wcstombs_s/swscanf_s) use the <real>_nid_postfix plain-definition form, per the feat(libs): Deathloop loading support (Ampr, NetCtlApDialog + stubs) #407 precedent, avoiding host interposition on Linux.
• Anything else (no DB hit, or DB name that does not recompute) falls back to collision-free UnknownNN exported through APS5_EXPORT.
• Every body stays a throwing stub (NotImplemented_nid_no_patch("NID"); return 0;), no real bodies.
• Emits a TechnicalDebt line per stub (unknown signature for real names, unknown name, signature for UnknownNN) with path/lib derived from --module (--td-path/--td-lib override, --td-out/--td-only for TD output); input documented as plain NID name CSV like aerolib.csv.
• PR description updated; git diff --check clean.

The open question on whether a generator like this belongs in tools/ remains for @boykopovar.

[oneandonlydean]

Thanks, this covers what I asked. Recomputing names are written as plain exports, C and C++ runtime names get the _nid_postfix form, the rest fall back to UnknownNN through APS5_EXPORT, and each stub gets a TechnicalDebt line.

Two small things:

1. compute_nid, SUFFIX and CHARSET are a copy of tools/nid_names.py. Importing them from there would keep a single definition of the NID scheme.
2. The default TechnicalDebt path for --module libkernel is core/libs/prx/libkernel/Export.cpp, which doesn't exist (libkernel's unknown stubs are in Unimplemented.cpp), so those links would be broken unless --td-path is given.

Whether the generator belongs in tools/ is still the open question from before.

[adriadam10]

Thanks for the rework. The two points from the last review are still open on the current head:

• compute_nid, SUFFIX and CHARSET (tools/stubgen.py:8-10, :77-88) are still a copy of tools/nid_names.py. Import them from there so the NID scheme has a single definition.
• The default TechnicalDebt path for --module libkernel (tools/stubgen.py:107-108) is core/libs/prx/libkernel/Export.cpp, which doesn't exist (libkernel's unknown stubs are in Unimplemented.cpp). Drop that special case or point it at the real file; the libc/libkernel branches of td_lib are the same as the default and can go too.

Also, a small thing: please add a sample run (input NIDs, DB lines and the generated output) to the PR description, including one NID that falls back to UnknownNN, so reviewers can check the output without running it.

@boykopovar: the open question from Dean's review stands: should this generator live in tools/ at all?

[Pouare514]

Thanks for the follow-up — both done, rebased onto current origin/main (f8c2f07).

1. NID scheme dedup: tools/stubgen.py no longer defines compute_nid/SUFFIX/CHARSET; it imports them from tools/nid_names.py (try: from tools.nid_names ... except ImportError: from nid_names ...), so both script (python tools/stubgen.py, cd tools && python stubgen.py) and module (python -m tools.stubgen, from tools import stubgen) runs work and stubgen.compute_nid is nid_names.compute_nid. Code output verified byte-identical on a sample DB; git diff --check clean.

2. libkernel TD path: default is now core/libs/prx/libkernel/Unimplemented.cpp (verified Export.cpp does not exist there). _default_td_path() checks Export.cpp, then Unimplemented.cpp, then src/Unimplemented.cpp on disk and falls back to Export.cpp, so other modules without a top-level Export.cpp (e.g. libSceAgc, libSceFont/src) also resolve correctly. --td-path/--td-lib overrides unchanged.

The tools/ placement question is still open for @boykopovar — no change made there.

[adriadam10]

Checked tools/stubgen.py on a synthetic DB with libkernel and libSceAmpr, as a script and with python -m tools.stubgen: real names plain, _/CRT names as _nid_postfix, unknown or non-recomputing NIDs as UnknownNN through APS5_EXPORT, and the libkernel TechnicalDebt path now points to Unimplemented.cpp. NID scheme is imported from tools/nid_names.py.

The conventions check was failing only on the unchecked Depends on: none; I ticked it in the PR body (no dependencies). Merging once CI is green.