Propagate repaired sealed tail into loglet to prevent read-stream freeze

[pcholakov]

Problem

Follow-up to #5049 (which handles recovery from a chain prefix-trim). This addresses the onset: a Bifrost replicated-loglet reader can be stranded in State::Reading on a sealed tail it can never reach, applying nothing — with no error and no log output — until the process is restarted. The partition keeps reporting Active and (for a leader) keeps accepting appends, so nothing obvious surfaces until traffic on that partition hangs.

Root cause

During a seal + reconfiguration race, a replicated loglet can be left sealed at a stale tail (notify_seal sets the sealed flag without advancing the offset; PeriodicTailChecker then exits on the sealed flag, and the local-sequencer find_tail keeps returning the stale watch). The background ReadStreamTask gates entirely on the loglet's known_global_tail watch — with a stale sealed tail it believes it is at the tail and parks forever.

The chain does learn the authoritative repaired tail during reconfiguration (check_chain → Decision::TailLsnSet), but the read stream only wrote it to a LogletWrapper field (set_tail_lsn) — a value the underlying ReadStreamTask never consults. So the one component that knew the true tail never told the component that was parked on the wrong one.

Fix

• Add Loglet::notify_known_tail (default no-op). ReplicatedLoglet forwards it to known_global_tail.notify(true, tail), which is monotonic — it can only advance the offset and set (never clear) the sealed bit, so it cannot regress a fresher view or unseal.
• Call it wherever the read stream observes a TailLsnSet: the Reading-state recheck (widened from Observe chain prefix-trim in Bifrost LogReadStream Reading state #5049's trim-only find_segment_for_lsn to the full check_chain), and the AwaitingReconfiguration / AwaitingOrSealChain arms. The parked task wakes, drains to the sealed tail, and hands off to the next segment.
• Rename trim_checked_version → chain_checked_version to match its widened role (it now gates the full chain recheck, not just trim scanning).

Test

follower_unfreezes_on_notify_known_tail (in the existing single-node run_in_test_env): builds a remote-sequencer ReplicatedLoglet reader (which genuinely parks — its watch starts Open(OLDEST), no PeriodicTailChecker), asserts it is parked, then calls notify_known_tail and asserts records drain under a bounded timeout. Reverting ReplicatedLoglet::notify_known_tail to the no-op default makes the post-notify read time out — so the test pins the production line, not just a call site. Also reading_state_propagates_repaired_tail_to_loglet pins the read-stream call sites.

Verification

• cargo nextest run -p restate-bifrost --all-features → 55/55.
• cargo clippy -p restate-bifrost --all-features --tests -- -D warnings → clean; cargo fmt clean.

Scope / follow-up

This is the propagation fix for the case where reconfiguration completes (the chain records a repaired tail). It intentionally does not touch PeriodicTailChecker / find_tail (the local-sequencer branch that trusts a frozen Sealed(stale) watch), nor add a bounded Reading-state safety-net timer — those cover the narrower case where reconfiguration never records a tail, and are tracked as a separate defense-in-depth follow-up.

Likely fixes #5033 (validated against the reporter's logs: all affected partitions had a completed chain reconfiguration whose repaired tail the reader never learned).

[github-actions]

Test Results

  8 files  ±0    8 suites  ±0   5m 5s ⏱️ +6s
 61 tests ±0   61 ✅ ±0  0 💤 ±0  0 ❌ ±0 
268 runs  ±0  268 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit ccc833d. ± Comparison against base commit 061157b.

♻️ This comment has been updated with latest results.