Return no_process from init_reader when the member is gone

[lukebakken]

Note

This PR was prepared by Claude (Anthropic's Claude Code) under the direction of @lukebakken, who reviewed the change before opening it. The fix was surfaced by a long-running high-throughput test. The code and analysis are AI-drafted and human-reviewed.

Problem

A 2-hour high-throughput test crashed a rabbit_stream_reader connection with exit:noproc raised from osiris:init_reader/4 when a consumer subscribed to a stream whose local member process had just gone (a retention resync restarted it in the same instant).

osiris:init_reader/4 and resolve_offset_spec/3 call osiris_util:get_reader_context/1, which does a bare gen:call/4 to the member process. When that process is not alive, gen:call exits with noproc. Neither function catches it, so the exit escapes: rabbit_stream_reader:init_reader/6 calls osiris:init_reader/4 inside a case that handles {ok, _} and {error, _} but not an exit, so the connection process crashes with a [error] gen_statem crash report, and its rabbit_stream_connection_sup then hits reached_max_restart_intensity. The condition is a benign race (the client reconnects and the run completes with no message loss), but it is loud in the logs and indistinguishable from a real fault.

This is inconsistent with how the same condition is handled elsewhere in osiris: osiris_writer:overview/1 and init_data_reader/3 guard get_reader_context/1 with is_process_alive/1 and return {error, no_process}, which osiris_replica_reader:init/1 handles as a clean stop. The two member-facing reader entry points in osiris.erl are the ones that neither guard the call nor catch the exit. init_reader/4's own -spec promises {ok, _} | {error, _}, so the leaked exit also violates the declared contract.

Full analysis in #235.

Solution

Wrap the get_reader_context/1 call in both init_reader/4 and resolve_offset_spec/3 and return {error, no_process} on a noproc exit, via a small shared reader_context/1 helper. This matches how osiris_writer already reports the same condition, and {error, no_process} is a shape rabbit_stream_reader:init_reader/6 already handles: it turns it into a STREAM_NOT_AVAILABLE response the client retries. {error, no_process} is added to both specs.

I catch both exit:noproc (a gen:call to a dead local pid) and exit:{noproc, _} (the tagged form a gen:call into a terminating gen_server can raise). The try covers only the get_reader_context/1 call, so errors from the downstream init_offset_reader/resolve_offset_spec are not masked. get_reader_context/1 and its other callers (get_stats/1, the two osiris_writer sites) are left unchanged: they either guard the call already or need the raw context, so narrowing the fix to the two unguarded reader entries avoids changing their behaviour.

This is the same class of benign, race-driven crash as #230 (fixed by #231) and #233 (fixed by #234): one code path handles the condition gracefully and a sibling path leaks an uncaught exception under retention churn. This fix is independent of those and was verified on a run with the #231 and #234 fixes both applied.

On testing

I have not added a test. The crash is an inherent race: to raise the real noproc the member pid must die in the window between the subscribe and the gen:call, so it is not reproducible on demand without mocking. The sibling guarded callers (osiris_writer:overview/1, init_data_reader/3) ship without dedicated tests for this condition, and #234 took the same approach for the analogous retention race. Happy to add a mock-based test if you would prefer one.

Closes #235

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request