Skip to content

GO-7561 concurrent AddLog of the same log returns ErrLogExists instead of WriteConflict - #115

Merged
requilence merged 4 commits into
mainfrom
go-7561-watch-missing-log
Oct 7, 2026
Merged

requilence merged 4 commits into
mainfrom
go-7561-watch-missing-log

Conversation

@requilence

@requilence requilence commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Every sync node that stores a space calls AddLog and then Watch when it loads the space, usually at the same time as the other replicas. On stage (2026-09-30 to 10-05) nodes logged failed to add consensus record: (WriteConflict) … WiredTigerIdIndex::_insert: index: _id_, each followed within 1 ms by received consensus error: log not found for the same space.

  1. Since v0.6.0 AddLog runs in a transaction. When several nodes create the same log at once, the losers fail with WriteConflict while the winner is still uncommitted. That error went out joined with ErrUnexpected, and drpc cannot read a code from errors.Join, so it reached the client with code 0 and the raw mongo text.
  2. The loser then watches the log before the winner commits and gets ErrLogNotFound. The client keeps its watcher registered and only logs the error, so the node got no ACL pushes for that space until it unloaded it or the watch stream reconnected.

Fix

  • db: tx runs a transaction again on TransientTransactionError (up to 10 attempts, 10 ms linear backoff), so a loser waits for the winner's commit and gets ErrLogExists; its watch then finds the log. Transactions return mongo errors as they are, so tx can see their labels; AddLog, AddRecord and DeleteLog then replace any uncoded error with ErrUnexpected (code 500), logging the original. AddLog and AddRecord no longer clear the payloads of their input, which a retried transaction needs again.
  • stream: a missing log is looked up once more. A watch can join a load that another watch started before the log was committed and share its ErrLogNotFound; ocache drops a failed load, so the second lookup reads the db again.
  • stream, defensive: a failed watch drops the log id before the error is sent. Before, the id stayed in the stream's watched ids although the stream was never attached, so a repeated WatchIds without UnwatchIds was skipped. consensusclient does not send that (UnwatchIds already removed the id), so this changes nothing for current clients; the error still ends the watch.

Remaining gap: if AddLog still fails (every retry exhausted), the node's watch ends with ErrLogNotFound and nodeSpace.AddConsensusError only logs it, as before. The node keeps its ACL and gets new records through headsync, without the push. A re-watch on the node would close this; not part of this PR.

An earlier version kept a watch on a missing log waiting for it. That was dropped: clients treat ErrLogNotFound as final, and the coordinator's acl watcher panicked on records after it.

Tests

  • stream: TestStream_WatchMissingLog checks the event order a client sees (second lookup, error ends the watch, re-watch, other errors); all four cases fail on main.
  • db: concurrent AddLog of the same log (fails on main with the stage WriteConflict), tx retry with injected labeled errors (retry, wrapped label, exhaustion, no retry of other errors), AddLog keeps its input, error conversion keeps code 500.

The db tests need MongoDB as a replica set; they were run against a single-node replica set.

Deploy

Requires anyproto/any-sync#805 in the coordinator, any-sync-filenode and any-sync-filenode2 first. consensusclient restores its watches after a reconnect outside its mutex; an UnWatch + Watch of the same log in that window, followed by the restore, sends the id twice on one stream. Before, the second request was skipped; now both get ErrLogNotFound, which an unpatched acl watcher turns into a double close. The window is narrow, and the same panic is reachable on main without this PR. Order: coordinator and filenodes → consensusnode → nodes (anyproto/any-sync-node#251, optional).

Known limits: retries are bounded by attempts, not time (a server-selection failure inside a transaction is also labeled transient); an unknown commit result is handled as before.

Part of GO-7561.

WatchIds added the id to the stream's watched ids before AddStream and kept
it when AddStream failed. The stream was never attached to the log, and a
later WatchIds for the id was skipped as already watched, so it stayed dead.

The id is now dropped before the error is sent; the error still ends that
watch, as clients expect. A node watches a log right after creating it, and
its lookup can share the ErrLogNotFound of a load another watch started
before the log was committed, so a missing log is looked up once more.
When several nodes create the same log at once, the losing AddLog
transactions failed with WriteConflict, which went to the client without an
rpc code, instead of ErrLogExists. tx now runs a transaction again on
TransientTransactionError, so a loser sees the committed log and gets
ErrLogExists.

Transactions return mongo errors as they are, so that tx can see their
labels; AddLog and AddRecord replace any non-consensus error with
ErrUnexpected afterwards, which keeps its rpc code. AddLog and AddRecord no
longer clear the payloads of their input, which a retried transaction needs
again.
Like AddLog and AddRecord, DeleteLog returned raw mongo errors, which reached
the client without an rpc code.
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

New Coverage 69.7% of statements
Patch Coverage 97.4% of changed statements (38/39)

Coverage provided by https://github.com/seriousben/go-patch-cover-action

@requilence requilence changed the title GO-7561 a watch that misses a log being created no longer stays dead GO-7561 concurrent AddLog of the same log returns ErrLogExists instead of WriteConflict Oct 5, 2026
consensusErr logged the original mongo error without the log id, while the
rpc request log only has the generic ErrUnexpected; the two could only be
matched by time.
@requilence
requilence merged commit 85a66eb into main Oct 7, 2026
6 checks passed
@requilence
requilence deleted the go-7561-watch-missing-log branch October 7, 2026 13:06
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 7, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants