Skip to content

Add low-level TLS state machine API (TlsContext / TlsSession)#130366

Open
wfurt wants to merge 14 commits into
dotnet:mainfrom
wfurt:TlsSession-clean
Open

Add low-level TLS state machine API (TlsContext / TlsSession)#130366
wfurt wants to merge 14 commits into
dotnet:mainfrom
wfurt:TlsSession-clean

Conversation

@wfurt

@wfurt wfurt commented Jul 8, 2026

Copy link
Copy Markdown
Member

Introduce a caller-driven, non-blocking TLS state machine API for System.Net.Security, enabling high-performance scenarios where the caller controls buffer management and I/O scheduling.

fixes #128871

supports Linux, macOS, Windows & FreeBSD

This is initial submit (already too big) to expose the approved API surface. I will work on cleanup & integration with SslStream once this lands.

Introduce a caller-driven, non-blocking TLS state machine API for
System.Net.Security, enabling high-performance scenarios where the
caller controls buffer management and I/O scheduling.

New public API surface (marked [Experimental("SYSLIB5007")]):
- TlsContext: immutable, shareable TLS configuration (certificates,
  protocols, ALPN). Created via CreateServer/CreateClient factories.
- TlsSession: per-connection TLS state machine with Encrypt/Decrypt
  and handshake operations. Supports buffer-based and socket-based modes.
- TlsBufferSession / TlsSocketSession: concrete session types for
  caller-managed buffers and OS socket fd binding respectively.
- TlsOperationStatus: enum indicating handshake/data operation results.

Platform support:
- Full implementation on OpenSSL (Linux, FreeBSD)
- Network.framework integration on macOS
- Stub (NotSupportedException) on Windows/other platforms

SslStream integration:
- Wire existing SslStream handshake through TlsSession on Unix platforms
  (wedge mode), reducing code duplication while preserving all existing
  behavior and passing all existing tests.

Native PAL additions:
- OpenSSL: BIO read/write helpers, SSL_set_retry_verify for deferred
  certificate validation, ClientHello peek/parse
- Apple: Network.framework server connection bootstrap for testing
@wfurt wfurt added this to the 11.0.0 milestone Jul 8, 2026
@wfurt wfurt requested review from a team and bartonjs July 8, 2026 17:40
@wfurt wfurt self-assigned this Jul 8, 2026
@wfurt wfurt requested a review from jeffhandley as a code owner July 8, 2026 17:40
Copilot AI review requested due to automatic review settings July 8, 2026 17:40
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new experimental, low-level TLS state machine API in System.Net.Security (TlsContext, TlsSession, TlsBufferSession, TlsSocketSession, TlsOperationStatus) and adds the native/PAL plumbing needed to support non-blocking, caller-driven TLS flows across OpenSSL, SChannel, and Apple Network.framework.

Changes:

  • Adds new experimental public API surface for a non-blocking TLS engine (TlsContext/TlsSession + buffer-driven and socket-bound session types).
  • Extends OpenSSL native shims to support fd-bound TLS (SSL_set_fd), handshake driving, retry-verify plumbing, and a socket BIO that can replay a prefetched ClientHello.
  • Updates Apple Network.framework PAL to support server-side TLS (identity wiring + server bootstrap), and adds/adjusts tests to cover the new surface.

Reviewed changes

Copilot reviewed 41 out of 42 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
src/native/libs/System.Security.Cryptography.Native/pal_ssl.h Adds OpenSSL shims for fd binding, raw handshake, and retry-verify control.
src/native/libs/System.Security.Cryptography.Native/pal_ssl.c Implements new OpenSSL shims; adjusts protocol support cleanup call.
src/native/libs/System.Security.Cryptography.Native/pal_bio.h Declares socket-replay BIO + TLS frame peek/prefix APIs.
src/native/libs/System.Security.Cryptography.Native/pal_bio.c Implements socket-replay BIO and TLS record peek support.
src/native/libs/System.Security.Cryptography.Native/opensslshim.h Updates OpenSSL shim function requirements and includes.
src/native/libs/System.Security.Cryptography.Native/entrypoints.c Exposes new BIO/SSL entrypoints to managed interop.
src/native/libs/System.Security.Cryptography.Native.Apple/pal_networkframework.m Adds server-side NW bootstrap flow + per-session queueing changes.
src/native/libs/System.Security.Cryptography.Native.Apple/pal_networkframework.h Updates NW create signature to accept server identity.
src/libraries/System.Net.Security/tests/UnitTests/System.Net.Security.Unit.Tests.csproj Links production OpenSSL options partial + fixes ReadWriteAdapter link path.
src/libraries/System.Net.Security/tests/UnitTests/Fakes/FakeSslStream.Implementation.cs Extends fake options to carry a remote cert validator hook.
src/libraries/System.Net.Security/tests/FunctionalTests/System.Net.Security.Tests.csproj Suppresses SYSLIB5007 in tests; adds new TlsSessionTests compile item.
src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamStreamToStreamTest.cs Adjusts skips/conditions for Network.framework behavior differences.
src/libraries/System.Net.Security/src/System/Net/Security/TlsSocketSession.cs Adds socket-bound, non-blocking TLS session wrapper type.
src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.Stub.cs Adds PNSE stub implementations for unsupported platforms.
src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.OpenSsl.cs Adds OpenSSL fd-mode fast paths and ClientHello peeking integration.
src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Implements core TLS state machine (buffered + socket-driven) and validation suspension flow.
src/libraries/System.Net.Security/src/System/Net/Security/TlsOperationStatus.cs Introduces the experimental operation status enum.
src/libraries/System.Net.Security/src/System/Net/Security/TlsContext.OpenSsl.cs Adds reusable OpenSSL SSL_CTX ownership/sharing for TlsContext.
src/libraries/System.Net.Security/src/System/Net/Security/TlsContext.cs Adds the experimental configuration type and context creation APIs.
src/libraries/System.Net.Security/src/System/Net/Security/TlsBufferSession.cs Adds buffer-driven session type exposing the core state machine surface.
src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.OSX.cs Adjusts macOS PAL routing and sync/async handling for Network.framework.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.TlsSessionWedge.cs Adds an initial “wedge” routing SslStream handshake steps through TlsSession.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs Integrates the wedge and refactors cert validation for reuse.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.NotUnix.cs Provides a stub wedge implementation for platforms not using it.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs Tweaks Apple async handshake setup and fallback behavior.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs Wires OpenSSL remote certificate validation delegate into options.
src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs Minor refactor of cached credential lookup key creation.
src/libraries/System.Net.Security/src/System/Net/Security/SslAuthenticationOptions.OpenSsl.cs Adds OpenSSL-specific per-session plumbing fields (SSL_CTX, fd binding, replay BIO/prefix).
src/libraries/System.Net.Security/src/System/Net/Security/SslAuthenticationOptions.cs Makes options partial; adds clone/copy helpers and OpenSSL validation hook state.
src/libraries/System.Net.Security/src/System/Net/Security/Pal.OSX/SafeDeleteNwContext.cs Improves NW transport EOF handling and disposal safety.
src/libraries/System.Net.Security/src/System/Net/Security/NetEventSource.Security.cs Generalizes cert validation event logging to accept non-SslStream senders.
src/libraries/System.Net.Security/src/System/Net/Security/LocalAppContextSwitches.cs Adds CaptureClientHello switch; adjusts NTLM default logic.
src/libraries/System.Net.Security/src/System.Net.Security.csproj Adds experimental warning suppression + compiles new TLS engine sources.
src/libraries/System.Net.Security/src/Resources/Strings.resx Adds new resource strings for TlsSession errors/platform support.
src/libraries/System.Net.Security/ref/System.Net.Security.csproj Adds ref dependency on System.Net.Sockets for SafeSocketHandle.
src/libraries/System.Net.Security/ref/System.Net.Security.cs Adds the new experimental public API surface to ref assembly.
src/libraries/Common/src/System/Experimentals.cs Reserves SYSLIB5007 for the new experimental TLS API.
src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs Adds P/Invokes for new OpenSSL SSL/BIO shims and new error code.
src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs Updates OpenSSL context caching/ownership and cert-verify callback behavior.
src/libraries/Common/src/Interop/OSX/Interop.NetworkFramework.Tls.cs Updates NW create signature to pass server identity.
docs/project/list-of-diagnostics.md Documents new SYSLIB5007 experimental diagnostic ID.

Comment thread src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Outdated
Comment thread src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Outdated
Comment thread src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Outdated
Comment thread src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Outdated
Comment thread src/libraries/System.Net.Security/src/System.Net.Security.csproj
Comment thread docs/project/list-of-diagnostics.md
Comment thread docs/project/list-of-diagnostics.md Outdated
Code fixes (all in TlsSession.cs):
- Remove debug 'CAPTURE-DEBUG' InvalidOperationException that was leaking
  a debug marker to consumers; silently skip ClientHello capture when the
  parse succeeds but the frame length doesn't fit (already benign).
- Fix ArrayPool leak in HandshakeSocketCore / ReadSocketCore: replace
  Array.Resize on the pool-rented _socketInBuf with a proper Rent/Copy/
  Return via new GrowSocketInBuf() helper. Prevents leaking the original
  pooled buffer and prevents returning a non-pooled resized array back
  to the pool on Dispose.
- Fix SetClientCertificateContext race with concurrent sessions: instead
  of disposing/nulling the shared TlsContext.CredentialsHandle, acquire a
  session-local credentials handle (mirroring the pattern in SetContext).
  ActiveCredentialsRef() already routes subsequent PAL calls through the
  session-local handle. Acquire eagerly so failures surface here.

Squash-regression restores:
- System.Net.Security.csproj: restore $(NetCoreAppCurrent)-openbsd TFM
  that was dropped from TargetFrameworks.
- System.Net.Security.csproj: fix stale ReadWriteAdapter.cs path — the
  file lives at $(CommonPath)System/ (not System/Net/).
- LocalAppContextSwitches.cs: restore IsOpenBsd flag used by
  NegotiateAuthenticationPal.ManagedSpnego on OpenBSD.
- docs/project/list-of-diagnostics.md: restore SYSLIB0065 row and update
  the SYSLIB5007 row to .NET 11 with the wording proposed by @vcsjones.
Comment thread src/libraries/System.Net.Security/src/System/Net/Security/TlsSession.cs Outdated
wfurt added 3 commits July 8, 2026 19:34
- Interop.OpenSsl.cs: fix indentation of the ~120 lines that got left at
  the previous indent level when the AllocateSslHandle body was wrapped
  in a try/finally block (comment from @MihaZupan).

- pal_ssl.c: drop the second argument from CryptoNative_EvpPkeyDestroy
  call (function signature is 1-arg on main). Fixes the linux-x64 native
  build (see CI 'dotnet-linker-tests (Build linux-x64 release Runtime_
  Release)' failure).
…SL API refactor

Reapply upstream 397e72c ("Prefer non-deprecated EC OSSL APIs where
possible") on top of the TlsSession-specific additions (BioGetPrefix,
BioNewSocketReplay, BioReadTlsFrame, SslDoHandshake, SslSetFd,
SslSetRetryVerify) after the recent upstream/main merge.
Copilot AI review requested due to automatic review settings July 9, 2026 00:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 42 changed files in this pull request and generated 4 comments.

- LocalAppContextSwitches.cs: restore IsOpenBsd branch in UseManagedNtlm
  defaultValue (was dropped in the squash; the IsOpenBsd field and its
  comment were restored in 51c03bc but the switch default was left
  inconsistent). Matches upstream/main.

- TlsSession.EnsureCredentialsAcquired: short-circuit when
  _sessionCredentialsHandle is already set (via SetContext or
  SetClientCertificateContext). Prevents allocating and publishing an
  unused shared TlsContext.CredentialsHandle that could race with other
  sessions on the same context. ActiveCredentialsRef() already routes the
  PAL to the session-local handle.

- TlsSession.Dispose: release _sessionCredentialsHandle. It was being
  acquired eagerly by SetContext/SetClientCertificateContext but never
  freed, so long-lived TlsContexts creating many sessions were leaking
  SafeFreeCredentials.

- System.Net.Security.Unit.Tests.csproj: fix stale ReadWriteAdapter.cs
  path — the file lives at $(CommonPath)System/, not System/Net/. Same
  fix was applied to src/System.Net.Security.csproj in 51c03bc but
  the unit test csproj was missed.

- JsonSourceGenerator.Parser.cs: mark AddTypeArgumentDiagnosticIds local
  function static (IDE0062). Unrelated regression pulled in from the
  recent upstream/main merge; blocks local libs build under
  TreatWarningsAsErrors.
Copilot AI review requested due to automatic review settings July 9, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 42 out of 43 changed files in this pull request and generated 6 comments.

Comment thread src/libraries/System.Net.Security/tests/FunctionalTests/TlsSessionTests.cs Outdated
wfurt added 2 commits July 9, 2026 03:19
Addresses @MihaZupan review feedback on PR dotnet#130366: 'replace these (and
similar) with ArrayBuffer to simplify all buffer management'.

Two sliding-window byte-buffer patterns migrated to the existing
System.Net.ArrayBuffer helper (already used by SslStream, HttpConnection,
Http2Connection, etc.):

- _pending / _pendingOffset / _pendingLength (staged TLS output drained
  by DrainPendingOutput and the socket send loop) -> _pendingBuffer.
  The AppendPending compact+grow+rent bookkeeping (~30 lines) collapses
  to EnsureAvailableSpace + CopyTo + Commit; DrainTo collapses to
  ActiveSpan slice + Discard.

- _socketInBuf / _socketInUsed (pre-fetched socket input consumed by
  HandshakeSocketCore / ReadSocketCore) -> _socketInBuffer. The custom
  GrowSocketInBuf helper and its manual BlockCopy compaction (~20
  lines) go away; growth is EnsureAvailableSpace and consume is Discard.

Net delta: -88 lines. All 10 target frameworks build clean; the existing
TlsSession functional test results are unchanged from HEAD (same 42/44
pass, same 2 pre-existing GetClientHelloBytes tests failing before and
after the refactor).
…o no-ops

Both ClientSession_GetClientHelloBytes_Throws and
ServerSession_GetClientHelloBytes_BeforeClientHello_Throws wrapped the
helper call inside a discard-typed lambda body:

  Assert.Throws<InvalidOperationException>(() =>
  {
      ReadOnlySpan<byte> _ = GetClientHelloBytesHelper(session);
  });

The C# compiler eliminates the entire assignment because the LHS is a
ref-struct local whose value is never read and cannot escape the scope,
compiling the lambda body to a single ret. The helper never runs, no
exception is thrown, and Assert.Throws fails with 'No exception was
thrown'.

Passing the helper call directly as the lambda body preserves the call
site (result is byte[], not a ref-struct discard) so the helper is
invoked, throws InvalidOperationException for len == 0, and the test
passes as intended.
wfurt added 3 commits July 9, 2026 03:40
The server-side ClientHello parse mutated the shared SslAuthenticationOptions
bag (_options.TargetHost = parsed.Value.ServerName) so parallel sessions
built from a deferred TlsContext could race on the SNI value. Since PAL
reads of TargetHost are all guarded by !isServer (OpenSSL, SChannel, Apple,
Android), that write was never needed by the handshake itself — it existed
only to (a) surface the resolved host via TargetHostName and (b) hand the
resolved host to ServerCertSelectionDelegate.

Introduce a session-local _sessionTargetHost field on TlsSession:

- Server sessions read/write _sessionTargetHost (default string.Empty until
  the first ClientHello is parsed).
- Client sessions continue to read _options.TargetHost (never mutated by us
  after SetContext).
- TargetHostName property routes on _context.IsServer.
- ResolveServerCertificateFromClientHello, HandshakeBufferedCore managed
  parse, and the OpenSSL native-peek path all target _sessionTargetHost.

This eliminates the largest silent-corruption path when a bootstrap
TlsContext is shared across concurrent server sessions (SNI-dispatching
front-end pattern). Removes one of the reasons _options.Clone() is called
per session; other reasons (CertificateContext mutation, PAL scratch
pockets on the bag) still require the clone.
Follow-up to the SNI-target-host change: the same "mutate the shared
options bag" pattern applied to CertificateContext. When a caller called
SetClientCertificateContext or when the server-side selector picked a
cert from ServerCertificateSelectionCallback, we wrote directly onto
_options.CertificateContext. That was safe today because _options is a
per-session clone, but it kept a session-scoped mutation on the bag we
would like to eventually share by reference.

Introduce session-local state on TlsSession:

- _sessionCertificateContext (SslStreamCertificateContext?): the effective
  cert context for this session. Initialized from _options at SetContext
  time; every mutation from SetClientCertificateContext and the server
  cert selector routes through SetSessionCertificateContext.

- _ownsSessionCertificateContext (bool): true only when the session
  itself constructed the context via SslStreamCertificateContext.Create
  in the server-selector path. Caller-provided contexts and the
  template context inherited from TlsContext are not owned by the
  session. Same convention as SslAuthenticationOptions.OwnsCertificateContext,
  scoped to the session.

- Private SessionCertificateContext get-only property used by all reads
  (LocalCertificate, HandshakeBufferedCore cert-resolution guard,
  ResolveServerCertificateFromClientHello guard).

- SetSessionCertificateContext(context, takeOwnership) helper releases any
  prior session-owned context (unless the new one is the same reference),
  installs the new value, and mirrors onto _options.CertificateContext
  (with OwnsCertificateContext=false) so the PAL sees the effective value
  without changing its signature. That mirror line is the single point that
  goes away when the PAL is later reshaped to consume the session directly
  — one of the pre-requisites to eliminating the per-session Clone().

- Dispose releases the session-owned context via ReleaseResources().
  _options.OwnsCertificateContext was flipped to false at SetContext, so
  _options.Dispose() no longer double-frees.

InitializeFromContext transfers CertificateContext ownership from the
options-clone to the session. Deferred SetContext (server SNI flow)
re-seats via SetSessionCertificateContext after CopyFrom so the new
per-tenant template value flows through the same code path. 44/44
TlsSessionTests still green across all 10 target frameworks.
Batch of drive-by fixes flagged by the latest Copilot review pass:

- TlsOperationStatus.DestinationTooSmall: expand XML doc. On buffered
  APIs the code means "destination span too small", on socket-bound APIs
  it means "socket returned WouldBlock mid-write". Doc now covers both.

- TlsSession.TargetHostName: honor the string? nullable contract. Getter
  now returns null when the underlying value is empty instead of
  surfacing "" and making callers unable to distinguish "not set" from
  "explicit empty string". Also adds an XML summary describing client vs
  server semantics.

- TlsSession.EnsureCredentialsAcquired: atomic install of the shared
  TlsContext.CredentialsHandle via Interlocked.CompareExchange. Multiple
  sessions on the same TlsContext could both see the field null, each
  AcquireCredentialsHandle, and overwrite/leak one another. Now the
  loser of the race disposes its own handle. Non-Windows PALs return
  null from AcquireCredentialsHandle so the CompareExchange is a no-op
  there.

- Interop.Ssl.cs: drop SetLastError=true from CryptoNative_SslSetFd and
  CryptoNative_SslDoHandshake. These shims report errors via out params
  and the OpenSSL error queue (SSL_get_error/ERR_get_error), never
  errno. Removes marshaller overhead for a value no caller reads.

- TlsSessionTests.MeasureHandshakeBytesAsync: drop the ArrayPool rent
  that was never used.

- TlsSessionTests.ServerSession_TlsResume_HonorsAllowTlsResumeOption:
  loosen the resumption savings threshold from < 0.6 to < 0.75.
  Observed on Alpine 3.24 x64 (Helix) with TLS 1.3: first=5504
  second=3544 -> 64.4% (test failed at 60% threshold). The absolute
  savings of ~1960 bytes are the omitted Certificate + CertVerify
  records, which is normal for a small test cert. TLS 1.2 still comfortably
  clears 75%; TLS 1.3 with small certs is closer to 60-70%, so 75%
  gives headroom without hiding real regressions.
…rverConnection

CreateServerConnection creates two dispatch_semaphore_t objects (inboundSem
and listenerReadySem) to synchronize with the listener state-changed and
new-connection handlers, but never releases them on any path. Each server
session leaked two semaphores.

Add dispatch_release for both semaphores at all six exit sites (five
error-return paths and the success path). Handlers that captured the
semaphores retain their own reference via block-copy, so it is safe to drop
our explicit reference here — the block references are released when the
listener releases its handler blocks (nw_listener_cancel + nw_release run
before our dispatch_release at every exit).

Flagged by Copilot on PR dotnet#130366.
Copilot AI review requested due to automatic review settings July 9, 2026 04:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 43 changed files in this pull request and generated 2 comments.

Comment on lines +774 to +783
private protected TlsOperationStatus HandshakeBufferedCore(
ReadOnlySpan<byte> input,
Span<byte> output,
out int bytesConsumed,
out int bytesWritten)
{
ThrowIfDisposed();
bytesConsumed = 0;
bytesWritten = 0;

Comment on lines 788 to +797
internal ProtocolToken NextMessage(ReadOnlySpan<byte> incomingBuffer, out int consumed)
{
if (TryNextMessageViaTlsSession(incomingBuffer, out ProtocolToken wedged, out consumed))
{
if (NetEventSource.Log.IsEnabled() && wedged.Failed)
{
NetEventSource.Error(this, $"Authentication failed. Status: {wedged.Status}, Exception message: {wedged.GetException()!.Message}");
}
return wedged;
}
…before SetContext

Before, TlsBufferSession.Handshake / TlsSocketSession.Handshake called on a
session whose TlsContext had not been assigned would flow past the disposed
check, deref _context! (the null-suppressing bang), and surface a
NullReferenceException from the first _context access downstream.

Add ThrowIfContextNotSet() right after ThrowIfDisposed() in
HandshakeBufferedCore and HandshakeSocketCore so the caller gets a
targeted InvalidOperationException with the existing
SR.net_ssl_tlssession_context_not_set message ("A TlsContext has not been
set on this session; call SetContext first.") instead.

Read/Write/Shutdown/RequestClientCertificate on the buffered surface already
gate on !_isHandshakeComplete before touching _context, so they already
throw InvalidOperationException (with a different message) prior to any
NRE opportunity; not changing them to avoid unrelated churn.

Flagged by Copilot on PR dotnet#130366.
Copilot AI review requested due to automatic review settings July 9, 2026 08:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 43 changed files in this pull request and generated 3 comments.

Comment on lines +472 to +475
if (cacheSslContext)
{
sslCtxHandle.TrySetSession(sslHandle, sslAuthenticationOptions.TargetHost);
}
Comment on lines +1029 to +1044
switch (status)
{
case TlsOperationStatus.Complete:
continue;
case TlsOperationStatus.NeedsCertificateValidation:
session.AcceptWithDefaultValidation();
continue;
case TlsOperationStatus.DestinationTooSmall:
DrainPending(session, socket, netOut);
continue;
case TlsOperationStatus.NeedMoreData:
inUsed += NonBlockingReceiveSome(socket, netIn, inUsed);
continue;
case TlsOperationStatus.Closed:
throw new IOException("Peer closed connection during handshake.");
}
Comment on lines +586 to +590
throw new InvalidOperationException("SetContext can only be called on a server-side session.");
}
if (!context.IsServer)
{
throw new ArgumentException("TlsContext must be server-side.", nameof(context));
SecureTransport on macOS pauses the client-side handshake with
errSSLServerAuthCompleted (mapped to SecurityStatusPalErrorCode.CertValidationNeeded)
BEFORE the handshake is complete on the wire, requiring an empty-input re-entry
into SSLHandshake after AcceptWithDefaultValidation / SetRemoteCertificateValidationResult
to produce ClientKeyExchange/Finished. On OpenSSL 3.0+ retry-verify the same
CertValidationNeeded status likewise pauses mid-handshake, whereas OpenSSL 1.1.x
and SChannel only surface the external-validation suspension via OnHandshakeCompleted
after _isHandshakeComplete is already true.

Two bugs blocked the macOS mid-handshake resume:

1. HandshakeBufferedCore's TLS-frame-header guard rejected the empty-input
   resume call with NeedMoreData, so the handshake never advanced. Fix: new
   _resumeAfterCertValidation flag, set in SetRemoteCertificateValidationResult
   when the PAL paused mid-handshake, consumed in HandshakeBufferedCore
   alongside _resumeAfterCredentials to bypass the frame-header guard.

2. On mid-handshake reject, no fault was surfaced (the existing post-hoc path
   only fires when _isHandshakeComplete is true). Set _externalValidationFault
   immediately so subsequent Write/Read throw AuthenticationException. Gate
   the top-of-HandshakeBufferedCore fault throw on
   _isHandshakeComplete || !_externalValidationResolved so the PAL can silently
   drive the accept-and-defer wire handshake to completion (matching OpenSSL
   1.1.x semantics) without the peer hanging waiting for our Finished.

Test skips: add SecureTransport-does-not-implement-TLS-1.3 guards to four
Tls13-parameterized / Tls13-only TlsSessionTests (mirrors the existing skip in
TwoSessions_HandshakeAndPingPong_InMemory_Succeeds):
 - SslStreamServer_RejectsClientCert_ClientObservesAlert
 - ServerSession_RemoteCertificateValidationCallback_IsInvokedPostHoc
 - ServerSession_ExternalValidation_RejectsClientCert_ServerFaultsPostHoc
 - SocketBoundSession_DeferredOptions_ProtocolMismatch_Fails

Results on osx-arm64 Debug:
 - TlsSessionTests class: 13 failing -> 0 failing, 333s -> 3.5s
 - Full System.Net.Security.Tests (5011 tests, non-outerloop): 0 failing

No changes to SslStream, SslStreamPal, or Apple PAL: all edits are inside
TlsSession, and SslStream on macOS keeps using its own #if TARGET_APPLE
VerifyRemoteCertificateAndGenerateNextToken path unchanged.

> [!NOTE]
> This commit was AI/Copilot-generated.
Copilot AI review requested due to automatic review settings July 9, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 43 changed files in this pull request and generated 3 comments.

Comment on lines +908 to +909
if (_clientHelloBytesBuffered is null)
{
Comment on lines +39 to +42
private partial bool TryNextMessageViaTlsSession(ReadOnlySpan<byte> incomingBuffer, out ProtocolToken token, out int consumed)
{
EnsureTlsSession();

Comment on lines +298 to +302
Interop.Ssl.SslErrorCode.SSL_ERROR_WANT_READ => TlsOperationStatus.NeedMoreData,
Interop.Ssl.SslErrorCode.SSL_ERROR_WANT_WRITE => TlsOperationStatus.DestinationTooSmall,
Interop.Ssl.SslErrorCode.SSL_ERROR_ZERO_RETURN => TlsOperationStatus.Closed,
_ => throw new AuthenticationException($"OpenSSL {op} failed: {error}"),
};
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[API Proposal]: low level TLS machine

5 participants