feat(ssl): run TLS 1.2 client connections over Mbed TLS

[loyalburrito]

What

sceSslConnect, sceSslRead and sceSslWrite no longer throw NotImplemented: they run a TLS 1.2 client over the Mbed TLS 3.6.7 submodule.

• Socket: the guest number resolves to the host socket through a new libkernel export, KernelLookupSocketNative_nid_no_patch. It checks libkernel's socket table, then the one libSceNet registers, so both socket() and sceNetSocket numbers work.
• Memory: session state comes from the context's existing memory pool and is returned on sceSslDeleteConnection. Each pool now has its own lock, so a handshake or read does not hold the library lock. A connection keeps its pool alive after sceSslTerm.
• Verification: the server is verified against the CAs loaded with sceSslLoadCert (chain, validity dates, host name, also sent as SNI). A loaded client certificate is sent when the server asks for one.
• Config: MbedConfig.hpp turns off MBEDTLS_SSL_PROTO_TLS1_3, and libSceSsl now links the mbedtls SSL library. SharpProspero's SceSslVersion stops at TLS 1.2, and Mbed TLS's TLS 1.3 needs process-wide PSA state that can't come from a context's pool.
• Debt: docs/dev/debt/libSceSsl.md records everything that throws or isn't verified on a console.

Questions for maintainers

Anything the sources don't settle throws for now:

1. No CA loaded. sceSslConnect throws, because no root bundle is shipped. Should a host bundle be trusted? feat(libkernel): serve the sandbox CA bundle from host TLS roots boykopovar/AnyPS5#5280 serves the host's CA bundle to the sandbox, which could fill this.
2. Verify callback. A non-null callback makes sceSslConnect throw. SharpProspero gives the arguments (an id, failure bits, the chain, its length, the user argument; a negative return fails the handshake), but not whether the id is the context or the connection, what the bits mean, what type the chain entries are, or when the callback runs.
3. Error codes. Are the console codes known for handshake, verification and socket failures, would-block, timeouts, or read/write before connect? All of these throw for now. Only 0x8095F006, 0x8095F004, 0x8095F007 and sharpemu's 0x8095F008 are known.
4. sceSslRead with flags 0. It returns what one record read delivers, the same as MsgFlagPartialRead (1). SharpProspero leaves open whether flags 0 waits to fill the buffer. Peek (3) throws.

Overlap

boykopovar#5911 (verify options, min version, ALPN), boykopovar#5953 and boykopovar#5969 (certificate readers; boykopovar#5969 also changes sceSslClose's signature) and boykopovar#6511 (certificate reader arguments) edit libSceSsl/Export.cpp and its debt entry, but none implements the handshake or I/O. Whichever merges second will need a rebase; I'm happy to do it.

Tested

Tested locally by the author on a machine with an NVIDIA GeForce RTX 3080 (12 GB).

Linux (Ubuntu 24.04, GCC 13), full Release build with BUILD_TESTING=ON, merged with current main:

• New guest_ssl_transport_kernel and guest_ssl_transport_net run an in-process Mbed TLS server on loopback that requires a client certificate, about 0.3 s each:
  • handshake, a write and read round trip, and an end-of-stream read that returns 0
  • four connections on one context in parallel threads
  • wrong host name, no CA loaded, a verify callback, a non-socket descriptor, a double connect, unsupported flags
  • pool memory returning to its baseline, and the socket staying open after delete
• Mutation checks:
  • With verification off, the wrong-host check fails.
  • With the pool lock removed, ThreadSanitizer reports data races in PoolCalloc.
• Sanitizers:
  • ASan+UBSan, leak detection on: clean on both transport tests, guest_ssl and guest_ssl_load.
  • ThreadSanitizer: clean on 3 runs of each transport test.
• Windows: not built with the WinLibs toolchain. The changed sources compile with -Wall -Wextra under Ubuntu's MinGW-w64 GCC 13 as a syntax and type check only.
• Full ctest: 1,281 of 1,289 pass. The 8 failures:
  • agc_driver_graphics, guest_datagram_* ×4, guest_named_shm_concurrency and guest_tcp fail the same way on unmodified main in that container (no IPv6, running as root, "VOP2 opcode is not implemented").
  • guest_kernel_fcntl aborted once under full-suite load. It then passed 30 runs alone and 3 loaded runs of all guest tests.
• python3 tools/check_conventions.py --base origin/main passes.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• GPUs tested: n/a (no AGC, shader recompiler or VideoOut changes); the author's local run was on an NVIDIA GeForce RTX 3080 12 GB
• AI-assisted: yes

🤖 Generated with Claude Code

https://claude.ai/code/session_01RoU3ZF76A6uR8sCA7NMLch