Notice when Team Collection monitoring stops working, and go Disconnected (BL-16729) by StephenMcConnel · Pull Request #8338 · BloomBooks/BloomDesktop

StephenMcConnel · 2026-09-09T23:15:14Z

Team Collection now notices when it can no longer see the shared folder, goes Disconnected, and tells the user — instead of carrying on looking normal while their teammates' work goes unseen.

The problem

Bloom watches the Team Collection's shared folder (Dropbox or a LAN share) with FileSystemWatchers. If that folder dropped out mid-session, nothing noticed: nobody subscribed to FileSystemWatcher.Error, and CheckConnection() only ran when the user did something (checkout, check-in, delete). Someone reading and editing their own checked-out book could go a whole session on stale data.

How it notices

Two routes, both funnelling into the existing disconnected state (yellow TC button, disconnected book-status panel, Reload Collection button):

Watcher Error — both repo watchers now subscribe. A dead watch disconnects immediately with no retry: .NET never re-establishes one, so even a returning folder would never produce another event.
ConnectionHeartbeat — re-checks every 60s. The only way to notice Dropbox has stopped syncing, where the folder is still there and we simply stop receiving other people's work. Owned by Start/StopMonitoring, so it is silent during SyncAtStartup, absent on a DisconnectedTeamCollection, and stops when we disconnect or dispose. One-shot re-arming makes overlapping ticks structurally impossible.

ConnectionFailureTracker holds a two-strikes rule (re-check after 15s; act only on a second failure of the same kind). The things CheckConnection looks at can lie — one dropped packet fails the dropbox.com probe, a Wi-Fi roam briefly kills GetIsNetworkAvailable() — and there is no automatic way back from a wrong disconnect.

How the user finds out

A persistent, non-modal toast (ToastService, the same mechanism as the existing TC clobber toast), because a recoloured top-bar button is easy to miss. Clicking it opens the TC dialog. Raised from NoticeConnectionProblem rather than MakeDisconnected, so it fires only for mid-session discoveries — not at startup (no workspace yet) or for subscription-tier disabling (wrong wording).

InternalBufferOverflowException is deliberately not a disconnect: the folder is reachable, we merely lost notifications. Buffers go to 64KB to make it rare; if it still happens the user gets a warning toast and the Reload button while staying connected.

Supporting changes

CheckConnection gains a quiet-probe overload. Without it, polling would append an un-deduplicated History message and raise a status-changed event on every tick of a healthy LAN-share session.
MakeDisconnected is now idempotent (returns false if already disconnected, so racing callers don't double-log or double-toast), stops the outgoing collection's watchers, and keeps it for Dispose. It previously nulled CurrentCollection without stopping it, and Dispose could then no longer reach it — so the abandoned collection went on watching and queueing changes forever.
PutBookInRepo / SetBookStatusString now clear _writeBookInProgress in a finally. A throw used to leave it set for the rest of the session, suppressing change notifications for that book — and would have silently killed the new heartbeat, in exactly the flaky-share case it exists to catch.
The three EnableRaisingEvents calls are guarded; BL-16679 was a crash from one.
CollectionsTabBookPane.tsx set disconnected where the interface field is isDisconnected, so a failed status fetch rendered the book as available for checkout.

Testing

15 new unit tests; full C# suite green. Not yet exercised in a running Bloom — nobody has pulled a share out from under a live Bloom or killed Dropbox mid-session. See the QA notes on the card.

Ref: https://issues.bloomlibrary.org/youtrack/issue/BL-16729

Devin review

This change is

…cted (BL-16729) Bloom watches the Team Collection's shared folder with FileSystemWatchers so it notices teammates' changes. If that folder dropped out mid-session, nothing noticed: nobody subscribed to FileSystemWatcher.Error, and CheckConnection() only ran when the user did something (checkout, check-in, delete). The user kept working, believing they saw the current state of the collection, while their teammates' work went unseen until Bloom was restarted. Bloom now notices by two routes, both funnelling into the existing disconnected state (yellow Team Collection button, disconnected book-status panel, Reload Collection button), plus a persistent toast, because a recoloured button is easy to miss: - Both repo watchers subscribe to Error. A dead watch disconnects immediately with no retry: .NET never re-establishes one, so even a returning folder would never produce another event. - A new ConnectionHeartbeat re-checks every 60 seconds. This is the only way to notice that Dropbox has stopped syncing, where the folder is still there and we simply stop receiving other people's work. It is owned by Start/StopMonitoring, so it is silent during SyncAtStartup, absent on a DisconnectedTeamCollection, and stops when we disconnect or dispose. ConnectionFailureTracker holds a two-strikes rule (re-check after 15s, act only on a second failure of the same kind), because the things CheckConnection looks at can lie and there is no automatic way back from a wrong disconnect. InternalBufferOverflowException is deliberately not a disconnect: the folder is reachable, we merely lost notifications. Buffers go to 64KB to make it rare, and if it still happens the user gets a warning toast and the Reload button while staying connected. Supporting changes: - CheckConnection gains a quiet-probe overload. Without it, polling would append an un-deduplicated History message and raise a status-changed event on every tick of a healthy LAN-share session. - MakeDisconnected is now idempotent (returns false if we were already disconnected, so racing callers don't double-log or double-toast), stops the outgoing collection's watchers, and keeps it for Dispose. Previously it nulled CurrentCollection without stopping it and Dispose could no longer reach it, so the abandoned collection went on watching and queueing changes forever. - PutBookInRepo and SetBookStatusString now clear _writeBookInProgress in a finally. A throw used to leave it set for the rest of the session, which suppressed change notifications for that book -- and would have silently killed the new heartbeat, in exactly the flaky-share case it exists to catch. - The three EnableRaisingEvents calls are guarded; BL-16679 was a crash from one. - CollectionsTabBookPane.tsx set `disconnected` where the interface field is `isDisconnected`, so a failed status fetch rendered the book as available for checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…llectionWhenDropboxStops

…artbeat (BL-16729) Two findings from Devin's review of PR #8338. Thread safety of the buffer-overflow path. HandleRepoWatcherError wrote to TeamCollectionMessageLog directly from a FileSystemWatcher callback. Both repo watchers can overflow at the same moment on different thread-pool threads, and the message log keeps one unsynchronized list which it enumerates to de-duplicate and then appends to -- while the UI reads that same list. Two simultaneous overflows, or one overflow during a UI read, could duplicate entries or throw. Worse, an exception escaping a watcher callback takes the process down. The overflow handling now goes to the UI thread the same way the disconnect path already does, and both watcher error handlers wrap their whole body so nothing can escape into the callback. Heartbeat integration was untested -- the existing tests exercised ConnectionFailureTracker's policy but never drove ConnectionHeartbeat.Tick, so the guards, the confirm-then-act sequencing, and disposal were all uncovered. Added eight tests that drive Tick directly with no timer, no network and no real repo: connection fine, one failure (waits), two in a row (disconnects), recovery in between (starts over), writing-to-repo and not-the-live-collection (skip and reset), post-dispose ticks are inert, and Start is a no-op under unit tests. That needed two seams on TeamCollection: IsLiveCollection (virtual, so a test can say whether this is the manager's current collection without standing up a live TeamCollectionManager) and ReportConnectionProblem (routing through the ITeamCollectionManager interface rather than the concrete TCManager, so it is mockable). Also documented what CheckConnection_QuietProbe_WritesNoMessages does not prove: the History writes it guards need a Dropbox-hosted repo, which a temp folder cannot reproduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…message-log locking (BL-16729) Three more findings from Devin's re-review of PR #8338. A probe that throws no longer preserves the previous failure. Tick's catch reported to Sentry and fell through, leaving the strike on the record -- so a failure, then a throwing probe, then another failure counted as two consecutive failures and would disconnect a collection that was never actually shown to be unreachable twice running. An exception tells us nothing either way, so it now breaks the run like any other skipped tick. The overflow warning no longer writes into an abandoned collection. If a racing watcher failure disconnected us while the warning was queued for the UI thread, MakeDisconnected had already swapped in a DisconnectedTeamCollection with its own message log, so the warning landed somewhere the status dialog no longer reads. HandleLostNotifications now checks IsLiveCollection before writing -- and "you may have missed some changes" is moot next to "you have lost contact with the collection" anyway. TeamCollectionMessageLog is now internally synchronized. Marshalling the overflow path to the UI thread (previous commit) closed the case Devin originally described, but not the general one: RunOnUiThreadLater runs inline when there is no window, and several API endpoints registered with handleOnUiThread false already reached WriteMessage from server threads via CheckConnection. Since WriteMessage enumerates Messages to de-duplicate and then appends to it, while the status properties enumerate the same list, an overlap could duplicate entries or throw InvalidOperationException. The check-and-append is now one atomic step and the status properties take the same lock. The status-changed event is deliberately raised outside that lock: it reaches WinForms and the websocket server, and holding a lock across that is how deadlocks happen. Not addressed: enumerating the public Messages list directly from outside the class is still unguarded. Nothing mutates it externally, and fixing it properly means returning snapshots rather than the live list -- a wider change than this branch should carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…d setup failures (BL-16729) MakeDisconnected now claims the transition atomically. Removing the earlier Interlocked gate (it could latch permanently and disable every future disconnect) left the guard non-atomic: callers arrive both directly, from a synchronous CheckConnection on a BloomServer thread, and indirectly, from a watcher or heartbeat failure marshalled onto the UI thread, so two could capture the same live collection, both pass the guard, and both go on to stop it and build a replacement. A short lock now covers just the claim -- read CurrentCollection, null it, set an in-progress flag -- and the rest runs outside the lock, since it writes to the message log and that raises an event reaching WinForms and the websocket server. The flag is cleared in a finally within the same synchronous method, so unlike the old gate it cannot latch. Added MakeDisconnected_ManyThreadsAtOnce_DisconnectsExactlyOnce, which asserts on the winner count and on message counts rather than only the resulting object, because the log de-duplicates Errors and would hide a double disconnect. TeamCollectionMessageLog.Messages is now a locked snapshot. The previous commit synchronized the log's own reads and writes, but callers still received the live list -- and teamCollection/getLog and teamCollection/logImportant are both registered with handleOnUiThread false, with HandleLogImportant enumerating it directly. A status change appending on the UI thread during one of those requests would throw "Collection was modified". The backing list is now private and every internal use is inside the lock. Only one production caller was reading the property, so this is contained. A connection problem raised with no current collection is no longer dropped silently. During ConnectToTeamCollection a brand-new collection is set up -- including StartMonitoring -- before being published as CurrentCollection, so a watcher that fails to start in that window had nowhere to report. It now goes to the log and to Sentry. Whether that case should disconnect outright is the open question already on the PR about collections we cannot watch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… (BL-16729) Same failure mode as the _writeBookInProgress bug fixed earlier on this branch, and the new heartbeat depends on this flag too. SyncAtStartup sets _syncIsRunning and clears it on its normal return and on its two explicit abort paths. Any other exception escapes to the broad catch in SynchronizeRepoAndLocal, which reports the problem and lets Bloom carry on monitoring -- with the flag still set. Since IsWritingToRepo consults it, the periodic connection check then skipped every tick for the rest of the session: Dropbox could stop later and nothing would ever look. Rather than re-indent a 570-line method into a try/finally, SyncAtStartup is now a thin wrapper that owns the flag in a finally and delegates the work to SyncAtStartupInternal. The inner method's existing assignments are left alone; they are now redundant but harmless, and leaving them keeps the diff to the actual fix. SyncAtStartup_Throws_StillClearsTheBusyFlag covers it, using Assert.Catch so the test fails rather than passes vacuously if the sync stops throwing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

StephenMcConnel · 2026-09-09T23:55:24Z

[Claude Opus 5 (1M context) from Steve McConnel's machine during preflight]

Consulted Devin on 2026-09-09, five times, most recently up to commit 2bacccab0.

One review per pushed commit. Devin accumulates findings across rounds and re-lists ones already fixed, so the count below is of distinct findings: 13 — 11 closed, 2 still open.

Fixed in code (8): watcher-callback thread safety; the untested heartbeat; probe exceptions preserving an outage strike; a stale overflow warning written into an abandoned collection; the non-atomic disconnect claim; message-log locking, then snapshots once Devin showed the readers really are off the UI thread; a silently dropped watcher failure during collection setup; and a failed sync leaving the heartbeat permanently switched off.

Closed with an answer rather than a change (3): the translate="no" flag is this repo's documented convention for new strings; the localization-priority question was in fact put to the developer during planning; and the useEffect rationale rule governs introducing an effect, not correcting one word inside a pre-existing one. Each has its reasoning on its own thread.

Still open, both facets of one question — should a Team Collection we cannot watch be treated as disconnected? Missing book watcher stays connected and New collection watcher failures ignored. The developer scoped that out when this work was planned, so reversing it is their call; both threads stay open until they answer.

Three of the fixes above are pre-existing bugs rather than anything this PR introduced — all three the same shape, a busy flag cleared only where the work finishes normally, and all three newly load-bearing because the heartbeat consults those flags.

Other reviewers: CI (pr-automation) passed at every commit. CodeRabbit is configured on this repo but has auto_review.enabled: false, so it does not review pushes.

Resolves the open review question about a Team Collection whose repo has no Books folder. The folder is created when a collection is set up, so the only way to be missing it is to be a joiner whose Dropbox has not delivered it yet. In that state there is nothing to miss: no teammate can have checked a book in. So this is deliberately NOT treated as a disconnection -- reporting one would be wrong and, for a first-time join, routine. What was genuinely broken is that we gave up for the whole session. The folder can arrive minutes later, and nothing looked again, so book changes went unseen until Bloom restarted. StartMonitoring now records that watching is deferred, and the periodic connection check retries: once the folder appears it starts the watcher and announces whatever is already sitting in it, since from the watcher's point of view those books are all new since Bloom started. The old early return also skipped the Other watcher, so a missing Books folder silently cost us collection-settings notifications too. Now only the books watcher is deferred. Considered and rejected: a single recursive watcher on the whole repo folder, which would notice the Books folder instantly and need no catch-up. One watcher means one NotifyFilter, so the Other folder would gain FileName/DirectoryName events it does not get today -- and that path reaches CheckWhetherRepoNowRequiresANewerBloom, which blocks in a modal and can shut the user out of the collection. It would also make the debounce helpers, which are per-watcher-per-event-type, share state between the two kinds of change, and add Lost and Found and our own Books/Temp write traffic to the buffer. That puts the risk on the path that works today for every collection, to fix a case that only affects a mid-sync joiner. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…6729) Devin's follow-up on the message-log locking: AfterMessageAdded appended to log.txt outside _messagesLock, so two threads writing at the same moment could reach disk in a different order than the in-memory list, and could collide over the file itself (RobustFile.AppendAllText retries IO errors but does not serialize callers). The append now happens inside the lock, alongside adding to the list, so the file ends up in the same order as memory and only one thread is ever appending. It is a short append; holding the lock across it costs little. The status-changed event still goes outside the lock, because that one reaches WinForms and the websocket server and holding a lock across it invites deadlock -- which is why the two were split in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

StephenMcConnel · 2026-09-10T15:41:36Z

[Claude Opus 5 (1M context) from Steve McConnel's machine during preflight]

Consulted Devin again on 2026-09-10, up to commit c956dda07 — a second preflight run after the developer answered the one open design question. (Earlier log: five consultations up to 2bacccab0.)

Distinct findings across the whole PR now 16, all 16 closed. New this run:

Message persistence remains unordered — real, and a flaw in my own earlier fix: I had split the message-log lock in the wrong place, leaving the disk append outside it, so the file could disagree with memory and two threads could collide over it. Fixed in c956dda07.
Deferred recovery bypasses change queue — verified rather than changed. The new recovery path calls HandleModifiedFile directly, exactly as StartMonitoringOnIdle already does for the same kind of catch-up, and both run on the UI thread.
Missing Books folder stays connected — not acted on. It re-opens the design question the developer settled this round, and the scenario it constructs is not silent: with Books missing, SyncAtStartup throws in GetBookList and the progress dialog reports the failure before monitoring ever reaches the deferred branch.

The two findings that were open awaiting the developer are now closed with their decision recorded on each thread: a Team Collection we cannot watch is not treated as a disconnection. Instead 08fd8cb60 has the periodic check retry, and start watching — and announce the books it finds — once Dropbox delivers the folder.

No review thread is left open. CI passed at every commit; CodeRabbit remains switched off on this repo (auto_review.enabled: false).

Found while working through a manual test report. RunOnUiThreadLater called Shell.GetShellOrOtherOpenForm() outside its try. That reaches Application.OpenForms, which is not thread-safe, and every caller of this method is on a file system watcher's thread or the heartbeat's. If it threw, the exception unwound past NoticeConnectionProblem and ReportConnectionProblem into ConnectionHeartbeat.Tick's catch, which reports to Sentry, resets the tracker and moves on -- so the disconnect was silently swallowed and Bloom went on looking connected. It is now inside the try, and a failure falls back to running the action inline, which is what the no-window case already did. Also log when the periodic check finds and then confirms a problem. Without this, somebody testing a real outage cannot tell "the check ran and decided" from "the check never ran at all" -- which is exactly the ambiguity that came up in testing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Devin's follow-up on the previous commit: making the failed OpenForms lookup fall back to running inline traded "the disconnect never happens" for "the disconnect happens on a watcher thread", which is not obviously the better bargain -- this work is marshalled precisely because it should not run there. Removed the form lookup instead. RunOnUiThreadLater now posts to Program.MainContext, the WinForms synchronization context captured once at startup, which is what ToastService already uses for its callbacks. That is better than either version: it never enumerates Application.OpenForms, so the thread-safety problem this started with cannot arise; Post is asynchronous even when called from the UI thread, which is a hard requirement because TryStartWatching calls in from the middle of StartMonitoring; and there is no form whose liveness has to be checked and which can be disposed between the check and the call. Neither remaining path runs UI work on a background thread. A null MainContext means no UI thread exists at all -- unit tests, or before Application.Run -- so nothing can be racing us and the action runs inline. If Post throws, the context is being torn down, so that reports to Sentry and deliberately does not fall back to inline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

StephenMcConnel · 2026-09-10T17:23:18Z

[Claude Opus 5 (1M context) from Steve McConnel's machine during preflight]

Consulted Devin up to commit a81463534. Distinct findings across the PR now 17, all 17 closed; the final round against this HEAD produced nothing new.

Two more since the last log entry, both from the developer's manual test of a lost drive:

UI work falls onto background threads — a fair catch on a fix I had just made. Recovering from a failed window lookup by running the work inline traded a missed notification for a data race, on work that is marshalled precisely so it does not run on a watcher thread. Fixed in a81463534 by removing the window lookup altogether: marshalling now posts to Program.MainContext, the synchronization context captured at startup, which never enumerates Application.OpenForms.
The fix that prompted it (a7cddff18) closed a real hole: that lookup sat outside its try, so if it threw, the disconnect was silently discarded and Bloom carried on looking connected — permanently, since every later check hit the same thing.

Also worth recording from that testing session: subst is not a valid way to simulate a lost share. Removing the drive letter does not invalidate already-open handles, so the watcher keeps working against the folder underneath and the instant-detection path never fires. The once-a-minute check still catches it, in about 75 seconds. The QA notes on the card have been corrected to separate the two mechanisms and to use a USB stick or a real mapped drive for the instant path.

No review thread is left open. CI passed at every commit; CodeRabbit remains switched off on this repo.

Manual testing (2026-09-10) disproved the justification I had written. The comment claimed the deferred case exists for a joiner whose Dropbox has not delivered the Books folder yet. You cannot join a collection in that state at all: the join fails earlier, in GetBookList, which throws on the missing folder. What is actually reachable is opening or reloading a collection already joined, at a moment when Books is absent -- Dropbox still restoring the shared folder onto a new machine, for instance. StartMonitoring runs on every collection open, not only at join. SyncAtStartup will have failed and told the user; what the retry avoids is having to restart Bloom once the folder finally arrives. Comment only; no behaviour change. Kept deliberately after weighing it: the failure mode is inert, and the same restructure stops a missing Books folder from silently skipping the Other watcher as well, which is a separate real gap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

StephenMcConnel · 2026-09-10T21:16:50Z

[Claude Opus 5 (1M context) from Steve McConnel's machine during pr-ready-for-human]

Consulted Devin on 2026-09-10 up to commit 6790f07ec — re-review clean, nothing new. That commit is comment-only (no code lines changed since a81463534, which Devin had already reviewed clean); the re-review was run anyway rather than assumed.

All 17 distinct findings across this PR remain closed, each with a documented outcome on its own thread. No review thread is open.

andrew-polk · 2026-09-11T18:01:25Z

@StephenMcConnel
This is a lot more complex than I was hoping.
It feels like too much for 6.5 to me.
What do you think?

StephenMcConnel and others added 3 commits September 9, 2026 17:11

Merge remote-tracking branch 'origin/Version6.5' into BL-16729-TeamCo…

815a595

…llectionWhenDropboxStops