GO-7556: recover from stale connections after sleep/wake

[requilence]

Supersedes #801. The pool part is redone: instead of per-peer generation stamps, Flush swaps in fresh caches. That is simpler, and it fixes a case #801 missed: a Get issued after Flush used to join a dial that started before the flush and wait it out on a dead connection.

After sleep/wake, clients kept reusing dead pre-sleep connections, so handshakes and RPCs hung for 10–30s or longer.

Changes

• pool:
  • Flush (new on pool.Pool/Service) atomically publishes a fresh incoming/outgoing cache pair. The Flush that replaced a pair walks it once after the swap and reports Closed for its peers before returning; watchers skip peers already reported, so each instance is reported exactly once, also with concurrent flushes. The old pair is torn down in the background, with at most 256 peer closes running at a time.
  • Lookups retry until the result comes from the current pair. The outgoing loader never dials for a replaced pair and dials under the pair's ctx. Get dials outgoing only on a real incoming miss, and waits out a busy incoming entry (for example a GC TryClose) instead of dialing a duplicate.
  • AddPeer returns ErrClosed during shutdown and waits for a mid-close entry before retrying. Retries are bounded, and flush swaps don't count against the bound. Accept bounds AddPeer at 10s.
  • Fast path via the new ocache.Peeker (Peek → miss/busy/hit, WaitClosing): cached lookups and misses allocate nothing. ocache.OCache is unchanged. ocache.PrometheusCollectors lets both cache pairs share one set of metrics; names and counting are unchanged.
  • peer.Peer implementations don't need to be comparable.
  • Incompatible-version backoff survives Flush. Close waits for pending teardowns, with a time limit.
• peer:
  • Sub-conn closes that used to block the caller (release after ctx cancel, failed handshake, gc) run in the background (closeAsync). Release and failed-handshake closes count toward the open limiter; gc closes don't. A waiting AcquireDrpcConn keeps one deadline across wake-ups.
  • Error change visible to callers: an RPC or stream (Invoke, NewStream, MsgSend/MsgRecv/CloseSend) cut short because its sub conn closed or the connection died now returns transport.ErrConnClosed (matches net.ErrClosed) instead of context.Canceled or drpc's "manager closed". A caller's own cancellation stays context.Canceled. Errors carrying a drpc code (server replies), application errors and a stream's io.EOF are never rewritten, so err == io.EOF still works.
• transport: yamux Open honours ctx. A dead connection is reported as ErrConnClosed via the new transport.NewConnClosedError, which keeps the original cause reachable. yamux reads keep plain io.EOF. A yamux write timeout counts only once the session has closed, because on a live session a slow uplink can cause one.
• peerservice: a dial cancelled by its ctx (for example by Flush) tries no further addresses and records no QUIC demotion outcome.
• handshake:
  • New OutgoingProtoHandshakeWithCloser hands the conn to a closer instead of closing it inline. OutgoingProtoHandshake behaves as before.
  • HandshakeError now unwraps, so errors.Is matches its cause (for example io.EOF or ctx errors).

Performance

Hit and miss paths in net/pool/pool_bench_test.go allocate nothing; on main, outgoing Get did 4 allocations. On hit paths the geomean is about −40% vs main. yamux Open with a cancellable ctx costs about 3µs and 4 allocs more per open, once per pooled sub conn. Servers never call Flush, so for them this branch brings faster lookups, bounded parallel closes on shutdown, and closes moved off caller paths.

Test plan

• go test -race ./..., with stress runs of the changed packages (-count=20, -cpu=1,4)
• Deterministic tests for the Flush/lookup/AddPeer/Close, observer, demotion, limiter and error-mapping races, checked against mutants
• anytype-heart and any-sync-node/coordinator/filenode/consensusnode build against this branch; heart's device/payments/peerstatus/recovery/space/filestorage tests pass
• Windows sleep/wake integration together with GO-7556: recover connections after sleep/wake; membership never stuck anytype-heart#3304

Linear: GO-7556

[github-actions]

New Coverage	63.2% of statements
Patch Coverage	93.7% of changed statements (569/607)

---

^{Coverage provided by https://github.com/seriousben/go-patch-cover-action}

[requilence]

Thanks for the recheck. Addressed in b672339.

Should fix

1. A dial cancelled by its ctx (for example by Flush) now stops at the current address and records no demotion outcome, so it never sets FallbackFailed. A real QUIC timeout followed by a successful yamux dial is still recorded. Covered by the new dialoutcome_test.go subtests: Flush mid-dial, plain cancel, and a later real fallback failure.
2. AddPeer: swaps no longer consume attempts, and the bound is attempts > retries, so it is reachable. Tests cover 3 and 5 swaps, and a two-duplicate replacement storm that ends with ErrExists after 3 evictions.

Worth raising

• yamux write timeout: kept unwrapped on a live session. In yamux v0.1.2, ErrConnectionWriteTimeout doesn't close the session. Its timer covers the wait on the shared send queue, so a slow but healthy uplink can trip it while other streams keep working (a test reproduces that). Mapping it to ErrConnClosed would mark healthy connections as dead; headsync for example matches net.ErrClosed. A session that is really stalled fails its keepalive, shuts down, and its streams already return ErrConnClosed. So it is wrapped only once the session is closed, and the comment says so.
• Get ctx: the redial loop now stops when its pair is replaced or the pool shuts down. Checking the caller ctx there turned out to be unreachable, because AfterFunc cancels the merged ctx asynchronously. Before the fix, the test saw dials into the old pair.
• Spin during Close: fixed by the same check, which returns ErrClosed.
• pick: no longer discards; eviction is left to the watcher.
• Incoming probe: Get uses Peek, so nothing loads into the incoming cache. waitClosing and the retries stay, because a mid-close or replaced entry still needs them.

Minor

• closeCaches now blocks, keepOnFlush is gone, abandoned() moved into export_test.go, and the comments are updated.
• Flush now walks the incoming cache once and the outgoing cache twice. Verdicts have to be read under swapMu before the fresh pair is published, while marking has to happen after the swap. That ordering also fixes a lost/duplicate Closed with concurrent Flushes: only the Flush that replaced a pair walks it.

go test -race ./... passes.