feat(libSceSsl): apply the verify options, run the verify callback, add ALPN, non-blocking sockets and sceSslClose

[frankea]

What

The follow-up oneandonlydean asked for on #7319 and loyalburrito reviewed on loyalburrito#3, on top of #6828's Session design, with that review's four points taken:

• Verify options. The handshake runs under MBEDTLS_SSL_VERIFY_OPTIONAL and sceSslConnect returns the first failure the connection's verify options select, with SharpProspero's SceSslVerifyOption bits and the codes KytyPS5's errno list gives: SSL_ERROR_UNKNOWN_CA 0x8095F00C, CN_CHECK 0x8095F00B, NOT_AFTER_CHECK 0x8095F00D, NOT_BEFORE_CHECK 0x8095F00E, INVALID_CERT 0x8095F00A for any other X.509 fault. The initial options are all five checks (0x3D), so a title that never calls sceSslEnableVerifyOption fails closed: with no CA loaded a default connection returns SSL_ERROR_UNKNOWN_CA instead of throwing. The result is final for the connection: a failure sends close_notify, a repeated sceSslConnect returns the same code, sceSslRead/sceSslWrite throw; a repeated sceSslConnect after success returns 0.
• Verify callback. Called once after the handshake, with the session lock released, with the connection id, the code above (0 when the chain passed), handles to a copy of the peer chain that sceSslGetSubjectName and the other readers on the connection's context accept, the chain length and the user argument. A negative return fails the connect with SSL_ERROR_APPLICATION_VERIFICATION_CALLBACK_CHECK (0x8095F0FD); any other return accepts the connection. The callback can use the connection (sceSslGetAlpnSelected, sceSslClose, sceSslDeleteConnection); a sceSslConnect from inside it, or from another thread while it runs, throws.
• ALPN and minimum version. sceSslSetAlpn's string is split on commas and offered; sceSslGetAlpnSelected returns the server's choice, null before the handshake or when nothing was selected. Every accepted SceSslVersion floors the handshake at TLS 1.2, all this build speaks.
• Non-blocking sockets. MBEDTLS_ERR_SSL_WANT_READ/WANT_WRITE return SSL_ERROR_WANT_POLLIN/WANT_POLLOUT (0x8095F013/14) from connect, read and write; the next sceSslConnect resumes the handshake; a write that blocks after sending part of the buffer returns the part sent.
• Reset and timeout, produced in the BIO (review point 2). Send/Receive map ECONNRESET/EPIPE and WSAECONNRESET/WSAECONNABORTED/WSAENETRESET to MBEDTLS_ERR_NET_CONN_RESET, which with a stream that ends without close_notify returns SSL_ERROR_SOCKET_CLOSED (0x8095170D). A blocking socket's expired receive timeout (EAGAIN on a socket without O_NONBLOCK, WSAETIMEDOUT on Windows) is told apart from a non-blocking would-block and returns SSL_ERROR_READ_TIMEOUT (0x80951710) from connect and read; the call can be repeated. An expired send timeout throws, as no write timeout code is known. A fatal alert returns SSL_ERROR_FATAL_ALERT (0x8095F010). When one of these ends the handshake the session is dropped and the next sceSslConnect starts over.
• sceSslClose. Sends close_notify once if the handshake completed, marks the connection closed (later connect, read and write throw) and leaves the socket to the title; returns 0 on a connection that never connected, already failed verification or was already closed.
• Debt entry (review points 1 and 3). The 0x3D default is recorded as unverified with the one piece of evidence (KytyPS5's HTTPS layer starts at ssl_flags 0xA7); the situation-to-code choices are listed as inferred from KytyPS5's names: WANT_POLLIN/WANT_POLLOUT rather than SSL_ERROR_EAGAIN, SOCKET_CLOSED rather than SSL_ERROR_EOF for a stream cut short, READ_TIMEOUT, FATAL_ALERT, INVALID_CERT as the catch-all, a repeated sceSslConnect returning the first result; and the callback's unknowns (which id, code or failed bits, element type, when it runs, whether a non-negative return overrides). The two title IDs the verify option entry named are removed; the entry text is otherwise rewritten around what the code now does.

Unchanged from #6828: the Session struct and its locks, the pool allocator, KernelLookupSocketNative_nid_no_patch, sceSslDeleteConnection (no close_notify, socket stays open), the throws for unknown flags, null buffers, descriptors that are not sockets and other handshake failures.

Tested

Windows 11 26200, WinLibs MinGW-w64 GCC 15.2.0 posix-seh ucrt r7, Release, BUILD_TESTING=ON, on f03a2da + this commit:

• guest_ssl_transport_kernel and guest_ssl_transport_net rewritten around the new behaviour, 5 runs each, all passing: a connection on a context with no CA and untouched options returns SSL_ERROR_UNKNOWN_CA twice, read throws, sceSslClose returns 0; with SERVER_VERIFY disabled the same connection returns 0 and reads to the server's close_notify; a verify callback on that context receives the connection id, SSL_ERROR_UNKNOWN_CA, one peer handle whose subject reads CN=localhost through sceSslGetSubjectName/sceSslGetNameEntryInfo, returns 0 and the connection becomes readable, and the peer handle throws once the connection is deleted; a certificate for another host returns SSL_ERROR_CN_CHECK twice, 0 with CN_CHECK disabled; a callback returning -1 on a passing chain gets code 0 and the connect returns SSL_ERROR_APPLICATION_VERIFICATION_CALLBACK_CHECK twice; SetMinSslVersion(TLS 1.2) + SetAlpn("h2,http/1.1") handshakes, GetAlpnSelected is http/1.1 after and null before, a second connect returns 0, sceSslClose returns 0 after the exchange, read and connect throw after it, a second close returns 0, pool memory returns and the socket stays open; a guest socket made non-blocking (SO_NBIO on libSceNet, the libkernel equivalent) handshakes through WANT_POLLIN/WANT_POLLOUT rounds, read with nothing pending returns WANT_POLLIN, then ping/pong and the close_notify arrive; a server that closes without close_notify ends the read with 0; a server that resets (linger 0) returns SSL_ERROR_SOCKET_CLOSED; on the libSceNet table a 200 ms SO_RCVTIMEO returns SSL_ERROR_READ_TIMEOUT from a read with nothing pending and the connection then exchanges ping/pong normally; sceSslClose on a connection that never connected returns 0; four parallel connections unchanged.
• guest_ssl, guest_ssl_load, guest_memory_pool_stats pass.
• python3 tools/check_conventions.py --base main passes.
• Not tested: Linux and macOS (the POSIX branches of HostReset/HostTimedOut and the test's close/fcntl paths are compiled only there), a send timeout, and a title.

Checklist

• Based on current main; no other open PR implements the same functions (feat(ssl): implement context-scoped PS5 certificate PEM reading #6511 adds sceSslGetPem in the same file and does not touch these calls)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in the technical debt file
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: PPSA01317 Gran Turismo 7 (PPSA01317 v01.680) on Linux/RADV: deterministic SIGSEGV in guest code (null deref r14+0x48) while the title's video-out/flip frame-pacing loop runs #6350 (the libSceSsl configuration calls GT7 makes before sceSslConnect)
• GPUs tested: n/a
• AI-assisted: yes (Claude Code)

[github-actions]

Progress report (376c040 → 51d0cb9)

📈 System libraries: 90.28% (+0.03%, +1 implemented functions)

✅ 1 implemented

Library	Name
libSceSsl	sceSslClose

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 6039	docs/dev/debt/libSceSsl.md

[oneandonlydean]

Linux CI (run 39151593422) fails guest_ssl_transport_kernel and guest_ssl_transport_net at "handshake before the server resets the connection". Windows passes both. It looks like a race in the new Serve::Reset case rather than a problem in Export.cpp.

The server resets the socket (SO_LINGER {1, 0} then close) as soon as its own mbedtls_ssl_handshake returns. At that moment the client can still be finishing its side of the handshake, reading the server's last flight. On Linux, an incoming RST discards data the client hasn't read yet, so the client's next recv returns ECONNRESET. Your new mapping then correctly turns that into SOCKET_CLOSED, so sceSslConnect(reset) doesn't return 0. On Windows the timing usually lets the client finish first.

Could the server wait until the client has finished its handshake before resetting? For example: after sceSslConnect succeeds, the client writes one byte; the server reads it and only then calls Abort. The sceSslRead(...) == ErrorSocketClosed check after that is still the behaviour you want to pin. If you'd like the handshake-time reset covered too, a separate case could expect sceSslConnect to return SOCKET_CLOSED there.

[frankea]

@oneandonlydean that reading is right, thanks. The server reset as soon as its own handshake returned, while the client could still be reading the last flight, and the RST discarding that unread data is a Linux behaviour the Windows run did not show. 51d0cb9 does what you describe: after sceSslConnect returns 0 the client writes one byte, the server reads it and only then closes with linger 0, and the sceSslRead(...) == SSL_ERROR_SOCKET_CLOSED check follows. I left the handshake-time reset out: with the server resetting at its own pace there is no point in the client's handshake it can reliably hit, so a case for it would be as timing-dependent as the one it replaces; the mapping it would pin (a reset during sceSslConnect returns SOCKET_CLOSED and drops the session) is the same Session::Code path the read case exercises.

On the 4090 box (Windows 11, WinLibs GCC 15.2.0, Release): guest_ssl_transport_kernel and guest_ssl_transport_net pass 5 runs each with the change, guest_ssl, guest_ssl_load and guest_memory_pool_stats pass. Linux still untested here, so the manual run is the check that counts.

AI-assisted: yes (Claude Code).