Skip to content

fix: fail fast when pushing to a transport being closed as unresponsive - #2709

Open
Anishishi wants to merge 1 commit into
eclipse-zenoh:mainfrom
Anishishi:fix-transport-fail-fast-close
Open

fix: fail fast when pushing to a transport being closed as unresponsive#2709
Anishishi wants to merge 1 commit into
eclipse-zenoh:mainfrom
Anishishi:fix-transport-fail-fast-close

Conversation

@Anishishi

@Anishishi Anishishi commented Jul 27, 2026

Copy link
Copy Markdown

Problem

When a peer's TCP connection stays established but the peer stops reading (zero-window; e.g. a frozen or abruptly power-cycled host), pushes of non droppable messages to that transport block for wait_before_close. When the deadline expires, TransportUnicastUniversal::handle_push_result logs Unable to push non droppable network message ... Closing transport! and spawns the close task on the RX runtime.

However, the transmission pipelines of the transport remain enabled until the close task actually runs. If the threads pushing to the dead transport are RX runtime workers (forwarding reliable data or declarations, holding routing locks), they immediately block again on the next message for another full wait_before_close each, starving the RX runtime. The close task never runs and the whole session is livelocked, logging the message above forever.

Fixes #1876, related to #2581.

Fix

Synchronously mark the transmission pipelines of the transport as disabled at the moment the closure is decided. This is a new TransmissionPipelineProducer::mark_disabled method that only performs an atomic store; contrary to disable it does not acquire the stage_in locks (which may be held by a currently blocked pusher).

push_network_message now fails fast with TransportClosed when the pipeline is disabled, before attempting to acquire the stage_in lock. Subsequent pushes to the unresponsive transport therefore return immediately, the RX workers are freed, and the spawned close task can run and perform the regular cleanup.

No behavior change for healthy transports: the check is a relaxed atomic load on the already-existing status flag.

Reproducer

A standalone reproducer with stock zenohd + examples (SIGSTOP-ed subscriber, blocking publishers, liveness probe) is attached to #1876. On current main the router wedges forever, alternating the error log between the RX workers every wait_before_close; with this PR it recovers right after wait_before_close.

Tests

  • New integration test unicast_unresponsive_peer simulates the zero-window peer with a stallable TCP proxy and asserts that, after the push failure that triggers the closure, the next push fails fast instead of blocking for another wait_before_close. It fails on current main and passes with this PR.
  • cargo test -p zenoh-transport --features transport_tcp passes.
  • Validated on a production multi-host rmw_zenoh deployment: 100 abrupt reboot cycles with zero occurrences of the wedge (previously reproduced within a few dozen cycles).

🏷️ Label-Based Checklist

Based on the labels applied to this PR, please complete these additional requirements:

Labels: bug

🐛 Bug Fix Requirements

Since this PR is labeled as a bug fix, please ensure:

  • Root cause documented - Explain what caused the bug in the PR description
  • Reproduction test added - Test that fails on main branch without the fix
  • Test passes with fix - The reproduction test passes with your changes
  • Regression prevention - Test will catch if this bug reoccurs in the future
  • Fix is minimal - Changes are focused only on fixing the bug
  • Related bugs checked - Verified no similar bugs exist in related code

Why this matters: Bugs without tests often reoccur.

Instructions:

  1. Check off items as you complete them (change - [ ] to - [x])
  2. The PR checklist CI will verify these are completed

This checklist updates automatically when labels change, but preserves your checked boxes.

When a non droppable message cannot be pushed within wait_before_close,
the transport is closed. However, the transmission pipelines remained
enabled until the close task actually ran, so every subsequent push to
the unresponsive transport blocked for another full wait_before_close
period.

This is especially harmful when the pushing threads are RX runtime
workers holding routing locks (e.g. forwarding reliable data or
declarations under the routing tables lock): they serially block on the
dead transport and starve the RX runtime, preventing the spawned close
task from ever running and deadlocking the whole session (eclipse-zenoh#1876, eclipse-zenoh#2581).

Synchronously mark the transmission pipelines as disabled when the
closure is initiated, so that subsequent pushes fail fast with
TransportClosed and the close task can proceed.

Signed-off-by: Anishishi <aki.nishi.work@gmail.com>
@Anishishi

Copy link
Copy Markdown
Author

Could someone with triage/write access please add the bug label to this PR?

I don't have permission to add labels myself, and the checklist CI (Generate / main) currently fails with:

PR must have at least one of the required labels: api-sync, breaking-change, bug, ci, dependencies, documentation, enhancement, new feature, internal

bug seems appropriate here. Thanks!

@Anishishi
Anishishi marked this pull request as ready for review July 27, 2026 03:06
@diogomatsubara diogomatsubara added the bug Something isn't working label Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blocking push can block the router indefinitely

2 participants