feat(ssl): apply the connection's verify options, minimum version and ALPN to the handshake

[frankea]

What

For boykopovar#6828 (upstream boykopovar#6828), the gap blawesom's rebase note named: main's verifyOptions, minimumVersion and alpn (boykopovar#5911) are recorded on the connection but Session::Configure never reads them. This wires them in and replaces three throws with the console's result codes, so a title that configures its connection the way GT7 does (sceSslEnableVerifyOption, sceSslSetMinSslVersion, sceSslSetAlpn, sceSslGetAlpnSelected, all imported by PPSA01317) gets answers instead of aborts.

• Verification. The handshake runs under MBEDTLS_SSL_VERIFY_OPTIONAL; afterwards sceSslConnect returns the first failure the connection's verify options select, using SharpProspero's SceSslVerifyOption bits (SERVER_VERIFY 0x1, CN_CHECK 0x4, NOT_AFTER_CHECK 0x8, NOT_BEFORE_CHECK 0x10, KNOWN_CA_CHECK 0x20) and the codes KytyPS5's errno list gives: SSL_ERROR_UNKNOWN_CA 0x8095F00C, CN_CHECK 0x8095F00B, NOT_AFTER_CHECK 0x8095F00D, NOT_BEFORE_CHECK 0x8095F00E, INVALID_CERT 0x8095F00A for any other X.509 fault. The "no CA loaded" throw goes: with the CA check on it is SSL_ERROR_UNKNOWN_CA, which is what a console with no matching root would say. A failed verification is final: close_notify is sent, a repeated sceSslConnect returns the same code, sceSslRead/sceSslWrite throw. A repeated sceSslConnect after success returns 0 again rather than throwing.
• ALPN. sceSslSetAlpn's string is split on commas and offered; sceSslGetAlpnSelected returns the server's choice, null before the handshake or when nothing was selected.
• Minimum version. Every accepted SceSslVersion floors the handshake at TLS 1.2, which is all this build speaks (1.3 is compiled out here), so the field is consumed without changing behaviour; the debt note says so.
• Socket results. MBEDTLS_ERR_SSL_WANT_READ/WANT_WRITE return SSL_ERROR_WANT_POLLIN/WANT_POLLOUT (0x8095F013/14) from connect, read and write, and the next sceSslConnect resumes the handshake instead of throwing "already connected"; a reset or an unannounced end of stream returns SSL_ERROR_SOCKET_CLOSED (0x8095170D), a timeout SSL_ERROR_READ_TIMEOUT (0x80951710), a fatal alert SSL_ERROR_FATAL_ALERT (0x8095F010). A write that blocks after sending part of the buffer returns the part sent. Everything else still throws through Fail as before.
• Left as it was, on purpose: the verify callback still throws (its argument semantics are the open question in your description), sceSslClose, and the initial verify options, which stay at 0 as main documents, so a title that never enables SERVER_VERIFY still gets an unverified session; a fail-closed default is a maintainer decision that waits on a root bundle (feat(libkernel): serve the sandbox CA bundle from host TLS roots boykopovar/AnyPS5#5280). Both are called out in the debt note.

Tested

Windows 11 26200, WinLibs MinGW-w64 GCC 15.2.0 posix-seh ucrt r7, Release, BUILD_TESTING=ON, on 91d8416 + this commit:

• guest_ssl_transport_kernel and guest_ssl_transport_net rewritten around the new behaviour and passing, 5 runs each: no CA with the initial options connects (unverified, documented); no CA with SERVER_VERIFY | KNOWN_CA_CHECK returns SSL_ERROR_UNKNOWN_CA twice and read throws; a certificate for another host returns SSL_ERROR_CN_CHECK with the checks on and 0 with CN_CHECK disabled; SetMinSslVersion(TLS 1.2) + SetAlpn("h2,http/1.1") handshakes and GetAlpnSelected returns http/1.1, null before the handshake and null on a connection without an ALPN list; a second sceSslConnect after success returns 0; the existing cases (verify callback throws, closed descriptor throws, unknown flags, peek, pool accounting, socket left open, four parallel connections) unchanged.
• guest_ssl, guest_ssl_load, guest_memory_pool_stats pass.
• python3 tools/check_conventions.py --base 91d8416fd passes.
• Not tested: Linux (the changed lines are platform-neutral; the Windows paths of Send/Receive are untouched), a non-blocking socket end to end (the WANT_* mapping is covered by code reading only; the test sockets are blocking), and a title.

Checklist

• Based on current main; no other open PR implements the same functions (this targets feat(ssl): run TLS 1.2 client connections over Mbed TLS boykopovar/AnyPS5#6828's branch; feat(libSceSsl): run the TLS handshake and I/O over the title's socket with Mbed TLS boykopovar/AnyPS5#7319 carries the same wiring on a different transport and is a draft that depends on feat(ssl): run TLS 1.2 client connections over Mbed TLS boykopovar/AnyPS5#6828)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in the technical debt file
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none (it is a change to feat(ssl): run TLS 1.2 client connections over Mbed TLS boykopovar/AnyPS5#6828's own branch)
• Unblocks: PPSA01317's libSceSsl configuration calls on feat(ssl): run TLS 1.2 client connections over Mbed TLS boykopovar/AnyPS5#6828
• GPUs tested: n/a
• AI-assisted: yes (Claude Code)

[loyalburrito]

Thanks @frankea, this is careful work, and most of it I'd like to take. I checked it against the sources and on Linux (the PR says Linux wasn't tested): cherry-picked onto the branch's current head 2edaf44a6, the code applies cleanly (only the debt text conflicts). The rewritten transport tests and guest_ssl, guest_ssl_load and the pool-stats test pass 3 runs in a row. ASan+UBSan is clean, and TSan shows nothing in 10 transport runs. All ten codes match KytyPS5's src/libs/errno.h at the pinned 4865dd9.

One blocker and three smaller points before merging:

1. Blocking: the initial verify options make verification fail open. main records the initial options as "unknown, taken as 0". That was harmless while nothing read them. Now that sceSslConnect reads them, 0 means no verification: any title that never calls sceSslEnableVerifyOption accepts any certificate from any server, and the "no CA" throw that used to stop this is gone. That includes ffmpeg.prx, the importer this entry is about. Today the branch fails closed, so this turns an unknown into a man-in-the-middle exposure. CONTRIBUTING's rule for unknowns points the other way, and the one source we have does too: KytyPS5's HTTPS layer at the same commit starts with ssl_flags = 0xA7 (network.cpp:191: SERVER_VERIFY | CLIENT_VERIFY | CN_CHECK | KNOWN_CA_CHECK | SNI). That's libSceHttp's default, not libSceSsl's, but it's evidence for verify-on, not 0. Could the initial options be all five checks (0x3D: SERVER_VERIFY, CN_CHECK, NOT_AFTER, NOT_BEFORE, KNOWN_CA), recorded in the debt entry as unverified on a console? A title that explicitly disables a check still gets exactly what your code gives it. With no CA loaded, a default connection would then return SSL_ERROR_UNKNOWN_CA instead of throwing, which is better than today.

2. Two of the described mappings can't happen with this transport.

• "A reset returns SSL_ERROR_SOCKET_CLOSED": Session::Receive/Send turn every host error other than EAGAIN/EINTR into MBEDTLS_ERR_NET_RECV_FAILED/SEND_FAILED, so MBEDTLS_ERR_NET_CONN_RESET is never produced and a reset still throws.
• "A timeout returns SSL_ERROR_READ_TIMEOUT": MBEDTLS_ERR_SSL_TIMEOUT only comes from a recv_timeout callback, and none is set. An expired libSceNet SO_RCVTIMEO reaches us as EAGAIN, so a blocking socket's timeout returns SSL_ERROR_WANT_POLLIN.

Either map ECONNRESET/EPIPE (and the Winsock equivalents) to MBEDTLS_ERR_NET_CONN_RESET in the BIO and tell a blocking socket's timeout apart from a non-blocking would-block, or drop those two cases and keep them as throws in the debt entry.

3. Record which code-for-which-situation choices are inferred. KytyPS5 gives names and values, not when the console returns them. The debt entry should say that the situation→code mapping is unverified, in particular:

• WANT_POLLIN/WANT_POLLOUT versus SSL_ERROR_EAGAIN (0x8095F00F), which is also in the table;
• SOCKET_CLOSED versus SSL_ERROR_EOF (0x8095177C) for a handshake cut short;
• INVALID_CERT as the catch-all;
• a repeated sceSslConnect returning 0 instead of an error.

4. Rebase onto 2edaf44a6. A maintainer merged main into the branch, including boykopovar#5969, so the debt entry conflicts.

With 1 changed and 2–3 settled, I'll merge it into the PR branch with your authorship.

---

Generated by Claude Code

[loyalburrito]

Update: boykopovar#6828 has been merged upstream (d704b80), so this branch is closed to new work and merging here would not reach upstream. Please open this against boykopovar/AnyPS5 main (or fold it into boykopovar#7319). The review points above still apply there. The code merges cleanly onto upstream main, with only the debt text conflicting.

---

Generated by Claude Code

[frankea]

Thanks for the review and the Linux and sanitizer runs; all four points are taken. Since boykopovar#6828 is in, this moves to a pull request against boykopovar/AnyPS5 main with the initial options at 0x3D, the reset and timeout cases either produced by the BIO or dropped back to throws, the inferred mappings listed as unverified in the debt entry, and the callback and sceSslClose that oneandonlydean asked for on boykopovar#7319. Closing this one.

AI-assisted: yes (Claude Code).