fix(server): let subscriber taps evict on full instead of wedging dispatch

[astrogilda]

Fixes #1136 (the runtime issue split out of #1101).

Problem

One undrained sink stalls EventQueueSource._dispatch_loop for every sink. The dispatcher's asyncio.gather awaits a blocking put on each sink, so a single full queue (default cap 1024) whose consumer went away blocks the gather forever. The incoming queue then fills, producers wedge in enqueue_event, and the task's event flow never recovers; see the task dumps and repro in #1101.

Change: split the sink semantics

The two kinds of sink want different fullness behavior.

The default sink is flow control. Its consumer drives task state, so a full queue that back-pressures the dispatcher (and, transitively, the producer) is correct, and it stays exactly as it is. A slow task store slows the pipeline down rather than losing events.

Tapped subscriber sinks are broadcast observers. Their consumers are remote and can be abandoned, most simply by a non-blocking message/send whose HTTP response has already returned. So tap() gains evict_on_full: bool = False. When an evict-on-full sink is full at delivery time, the dispatcher force-closes it (close(immediate=True), warning logged) and it detaches through the existing remove_sink path; dispatch to the remaining sinks continues, and any producer parked on the incoming queue recovers. The consumer of an evicted sink sees QueueShutDown on its next dequeue, same as any closed queue. There is no new exception type and no API surface beyond the keyword.

The test is fullness at delivery time and nothing else. An abandoned consumer is the case that motivated this, but a consumer that is alive and reading and has simply fallen max_queue_size events behind hits the same predicate and is evicted the same way. At the production default that is 1024 events behind, and at max_queue_size=2 a consumer sleeping five milliseconds between reads is evicted after draining one event. The warning text and the _deliver_to_sink docstring currently say the consumer "is not draining it" and call it an abandoned subscriber, which claims something the check never tested, and a reword is coming that names the queue bound instead and makes no claim about the consumer.

A sink that already closed since the dispatch snapshot was taken is skipped before the fullness check, so a graceful close(immediate=False) whose consumer is still draining is never upgraded to immediate=True and keeps its trailing events.

The full() pre-check is race-free: the dispatcher is the only writer to a sink queue and consumers only read, so a queue observed non-full cannot become full before the put.

tap() defaults to False, so the documented blocking contract is unchanged for existing callers. ActiveTask's subscriber tap opts in explicitly, and that is the one production site that wedges, so the fix reaches existing deployments without changing the default of the public primitive.

Tests

Five tests. test_evict_on_full_sink_is_evicted_and_dispatch_continues puts a full capacity-1 evict sink through dispatch and asserts it is closed and detached while the default sink still receives every event. test_producer_unblocked_after_evict_on_full is the #1101 runtime shape in miniature: producers blocked on a full incoming queue recover once the sink is evicted. test_default_tap_keeps_backpressure pins the existing flow-control contract, a default tap still blocking dispatch when full. test_graceful_close_not_upgraded_to_immediate_by_eviction covers the skip above, a gracefully closing full sink keeping its trailing events instead of being force-closed. test_subscriber_taps_are_evict_on_full asserts that ActiveTask.subscribe taps its subscriber sinks with the keyword set.

The eviction and producer-unblock tests fail without the fix. Full suite: 1761 passed, 90 skipped, 3 xfailed, 1 xpassed.

Out of scope

The deeper root cause is subscriber-sink lifecycle: a sink tapped for a consumer that has gone away should be closed by its owner at connection teardown, not discovered full later. That needs its own look at the request-handler layer. This PR keeps the dispatch layer from letting any such sink take the task down in the meantime.

Edited 2026-08-20 to say what the eviction check actually tests. The description read as though only an abandoned consumer is evicted; the predicate is queue fullness, so a live consumer more than max_queue_size events behind is evicted too, and the warning wording that claims otherwise is being reworded.

[gemini-code-assist]

Code Review

This pull request introduces an evict_on_full mechanism for event queue sinks to prevent abandoned remote subscribers from blocking the dispatcher and wedging the entire event pipeline. Sinks configured with this option are force-closed and detached when full, while default sinks retain their standard backpressure behavior. The changes are well-covered by new unit tests. The review feedback highlights a potential race condition in _deliver_to_sink where a gracefully closing sink might be prematurely force-closed if it is full, and suggests adding a sink.is_closed() check to avoid this issue.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[github-actions]

🧪 Code Coverage (vs main)

⬇️ Download Full Report

	Base	PR	Delta
src/a2a/server/events/event_queue_v2.py	91.28%	91.83%	🟢 +0.54%
Total	92.97%	92.98%	🟢 +0.01%

Generated by coverage-comment.yml

[astrogilda]

@rohityan Quick state of the PR so the review is cheap to pick up. The branch is rebased onto current main (pushed the 17th) and the full suite is green: 1761 passed, 90 skipped, 3 xfailed, 1 xpassed. Five regression tests pin the behavior, including the two that fail without the fix and one that pins the existing backpressure contract for default sinks, so nothing changes for current callers. The one review finding so far, gemini-code-assist's graceful-close race, is fixed: _deliver_to_sink now skips a sink that reports is_closed(), with its own regression test, so nothing is pending on my side.

For prioritization context, this fixes a permanent runtime deadlock. One abandoned subscriber sink wedges dispatch for the whole task and event flow never recovers (#1136, task dumps and repro in #1101). The diff is deliberately narrow: one keyword on tap(), default unchanged, and a single opt-in at the one production site that wedges. If anything would make the review easier, like splitting the tap() primitive change from the ActiveTask opt-in, a different name for the keyword, or more test coverage somewhere, say the word and I'll turn it around same day.

[astrogilda]

@rohityan This one is also rebased onto current main and green (events plus agent_execution, 156 passed, ruff clean). It has been ready since the 17th, and #1136 is assigned here. Same shape as #1105: small fix, behavior we already agreed on, just needs a look whenever you have the bandwidth.

[astrogilda]

@rohityan Rebased again now that #1105 is merged. Its tests landed in the same file, so the branch drifted to a conflict; cleared now and still green (events plus agent_execution, 114 passed, ruff clean). Independent of #1105 and ready whenever you can take a look.

[astrogilda]

Rebased onto current main (cff6727) and the full matrix is green again on 83fdd8d, so this is mergeable on sight rather than needing anyone to resolve anything first.

@ishymko @sokoliva @holtskinner — tagging you rather than pinging the issue thread again, since you three are the ones landing changes here at the moment and I would rather ask the right people once than the same person a fourth time. Genuinely no urgency implied; if the shape is wrong I would rather hear that than have it sit.

What it is, briefly, since it has been open a while. EventQueueSource dispatch wedges when one subscriber tap stops draining: the dispatcher blocks on the full tap, every other subscriber stops receiving, and nothing recovers. This makes the full tap evict rather than block, so a single slow or abandoned consumer degrades only itself. That is issue #1136, which @rohityan confirmed the direction on and assigned to me pending this landing.

Two commits, four files, and the tests are the part worth reviewing: they fail without the change and cover the eviction path and the closed-sink case rather than only the happy path.

One thing I would flag rather than leave for review to discover. This is adjacent to #1105, which landed in the same file, so the tests sit next to the ones that came with it. If you would prefer these squashed into one commit, or the eviction policy made configurable rather than fixed, both are easy changes and I would rather make them than argue for what is here.

[astrogilda]

Correcting my own tagging above, since I picked those names by counting recent merges rather than by looking at what each person merges.

@mykytanetipa — this is really one for you. You reviewed and merged #1105 in this same file, and your review comment there is the standard this PR is trying to meet: dispatcher tasks cancelled but not awaited, surfacing as "Task was destroyed but it is pending". The five regression tests here pin exactly that property on the dispatch path rather than the close path. If the eviction policy is the wrong call I would rather hear it from you than have it sit.

One practical note: #1134 touches event_queue_v2.py too. If that lands first I will rebase the same day and say so here, rather than leaving you to discover the conflict.

[astrogilda]

There is a defect in my own change here, and I would sooner flag it than leave it for review to find. The warning this logs says the sink's consumer "is not draining it". What _deliver_to_sink actually tests is sink._evict_on_full and sink.queue.full(), and full is true for any consumer that has fallen max_queue_size events behind, whether it stopped reading or is merely slower than the producer. The warning asserts something the predicate never checked.

I built the case that separates the two. Tap a sink at max_queue_size=2, give it a consumer that reads every event and sleeps five milliseconds between reads, then enqueue ten events. The sink is evicted and closed after that consumer has drained exactly one event, and it was alive and reading throughout. In production subscribe() taps with the default bound, so the threshold is 1024 events behind and not two, but the predicate is the same one. Someone reading that warning at three in the morning goes hunting for a dead consumer and finds a live one.

So I will push a reword instead of defending the wording. The warning should say the sink's queue is full and name the bound it hit, with no claim about what the consumer is doing. The docstring is the softer case, because evict_on_full is opt-in and subscribe() sets it under a comment about remote consumers that can be abandoned. Even there, "an abandoned subscriber" describes the caller's expectation and not the test, so I would have it say a consumer more than max_queue_size events behind. If you would prefer the eviction threshold be configurable per tap instead of pinned to the queue bound, that is also a small change, and I would sooner make it than argue for what is already here.

The same run confirms what the change exists for. The producer never blocked. Ten events went in under half a millisecond, and a healthy sibling sink drained all ten while the slow one was being closed. Without the change the dispatcher parks on the full sink inside its gather, every other subscriber stops receiving, the incoming queue fills behind it, and producers wedge in enqueue_event with nothing to recover them.

On the policy, closing the sink is the right eviction and dropping events would not be. A closed sink surfaces to its consumer as QueueShutDown out of dequeue_event, which is what I saw in that run, so the subscriber observes a terminated stream and not a stream that quietly skipped something, and it can come back through a SubscribeToTaskRequest. Dropping the oldest event leaves a hole no client can detect. That is the worse of the two failures even though it looks like the gentler one.

State in case it helps with scheduling: fifteen checks green at 83fdd8d across Python 3.10 through 3.14, and git merge-tree --write-tree against main at 4e71245 exits clean, three commits behind, no conflict. #1134 has not landed, so the same day rebase I promised above is not triggered.

@mykytanetipa whenever you have time, and no urgency. If the eviction policy is the wrong call for this queue I would rather hear that than have it sit.

Sankalp

[mykytanetipa]

Correct, well documented fix. Confirm that eviction policy is a correct approach here.

One follow-up I think can land separately, not blocking this PR: eviction currently surfaces to the subscriber as QueueShutDown, which is indistinguishable from normal stream completion -> a slow client may get dropped mid-task with no error signal.

Approved