Skip to content

servlet: fix UndertowTransportTest flakiness by eliminating container deadlocks#12908

Draft
AgraVator wants to merge 3 commits into
grpc:masterfrom
AgraVator:fix-undertow-test-flakiness-12778
Draft

servlet: fix UndertowTransportTest flakiness by eliminating container deadlocks#12908
AgraVator wants to merge 3 commits into
grpc:masterfrom
AgraVator:fix-undertow-test-flakiness-12778

Conversation

@AgraVator

Copy link
Copy Markdown
Contributor

Background & Problem Statement

UndertowTransportTest.basicStream intermittently failed in CI with:

java.lang.AssertionError: message expected
    at io.grpc.internal.AbstractTransportTest.basicStream(AbstractTransportTest.java:917)

Root Cause

AsyncServletOutputStreamWriter previously assumed that readyAndDrained could only remain true while onWritePossible() was actively executing.

However, when written payloads fit inside the servlet output buffer, outputStream.isReady() remains true after runOrBuffer() finishes, so runOrBuffer() exits leaving readyAndDrained == true.

When Undertow subsequently dispatched WriteListener.onWritePossible() after async I/O write completion:

  1. onWritePossible() saw readyAndDrained == true.
  2. It called assureReadyAndDrainedTurnsFalse(), which parked the Undertow container worker thread via LockSupport.parkNanos(TimeUnit.MINUTES.toNanos(1)).
  3. Because the application thread had already returned from runOrBuffer(), no thread was running to clear readyAndDrained, leaving Undertow blocked for 60 seconds.
  4. Meanwhile, AbstractTransportTest.basicStream() timed out after 10 seconds waiting at clientStreamListener.messageQueue.poll(), triggering AssertionError: message expected.

Solution

  1. Removed assureReadyAndDrainedTurnsFalse() and LockSupport parking:
    A state of readyAndDrained == true with an empty writeChain and isReady() == true is a valid quiescent state; onWritePossible() verifies the queue is empty and exits cleanly without parking worker threads.
  2. Simplified CAS updates in runOrBuffer():
    Updated state cleanup when outputStream becomes unready (isReady() == false), resolving concurrency races caught by Lincheck model checking.
  3. Unit Tests:
    Added AsyncServletOutputStreamWriterTest verifying non-blocking execution of onWritePossible().

Fixes #12778

… thread parking deadlocks (grpc#12778)

Fixes a long-standing flakiness bug in UndertowTransportTest.basicStream
where the container worker thread would park for up to 1 minute in
AsyncServletOutputStreamWriter.assureReadyAndDrainedTurnsFalse(), causing
the test listener poll to time out after 10 seconds and fail with
AssertionError: message expected.

Root Cause:
AsyncServletOutputStreamWriter previously assumed that readyAndDrained could
only be true while onWritePossible() was actively executing and that any call
to runOrBuffer() would return readyAndDrained back to false. However, when written
payloads fit inside the servlet output buffer, outputStream.isReady() remains
true after runOrBuffer() finishes, so runOrBuffer() exits leaving
readyAndDrained == true.

When Undertow subsequently dispatched WriteListener.onWritePossible() after async
I/O write completion, onWritePossible() saw readyAndDrained == true and invoked
assureReadyAndDrainedTurnsFalse(), parking the Undertow container thread via
LockSupport.parkNanos(1 min). Because the application thread had already returned,
no thread was running runOrBuffer() to clear readyAndDrained, leaving Undertow
blocked for 60 seconds and causing the test timeout.

Fix:
1. Removed assureReadyAndDrainedTurnsFalse() and LockSupport parking logic.
   A state of readyAndDrained == true with an empty writeChain and isReady() == true
   is a valid quiescent state; onWritePossible() verifies the queue is empty and
   exits cleanly without parking.
2. Safe state updates in runOrBuffer() when outputStream is not ready.

Fixes grpc#12778
@AgraVator AgraVator closed this Jul 13, 2026
@AgraVator AgraVator reopened this Jul 13, 2026
@AgraVator AgraVator marked this pull request as draft July 13, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

servlet: UndertowTransportTest.basicStream is flaky

1 participant