Skip to content

[raft] Improve startup and joining procedure #1802

Open

Description

@MariemBaccari

Following feedback on #1798 and #1780, design a startup and join procedure that is k8s friendly and avoids paths where trivial mistakes lead to a silently failing cluster.

Design proposal according to the feedback

  • Every node has a unique name within the cluster (use the public endpoint)
  • The name is replicated so that the node can still be uniquely identifiable within the cluster
  • It has 3 Raft IDs (one per service)
  • The IDs are autogenerated at the start and saved in the WAL's metadata

Starting up and handling membership

Bootstrapping a cluster

  • Every node starts with its own URL only
  • Nodes don't find a wal, they generate an ID per service
  • They wait
  • User runs dssctl bootstrap --node A (new admin endpoint)
  • Node A runs raft.StartNode and creates a new cluster with itself as a leader
  • Add the other nodes one by one to the cluster with dssctl add

Restarting a node

  • The node finds its old wal and restarts normally

New node joining and existing cluster

  • Start node then add it with dssctl add like the last step of the bootstrap procedure

Resetting a node that has lost or corrupted its datadir

  • Delete the datadir
  • Node starts, generates a new ID and waits
  • Old ID can be removed and the new one added

(This design removes all state-dependent flags and ensures that, either nodes know what to do at startup or that they will wait for a signal on the admin API from dssctl)

Activity

  1. added
    raftstoreRelating to the application-layer consensus implementation based on raft
    on Oct 22, 2026
  2. mickmis commented on Oct 22, 2026

    @mickmis
    Contributor

    Approach LGTM in general.
    FTR I had a look at the CRDB documentation to see how they manage that, as I think that in general it should be a good inspiration on how to handle the operations. How they do it is that they start the nodes with passed as argument (--join) several peers (they recommend 3 to 5), which are used only for the initial connection to the cluster. The actual list of peers is then different. This has the advantage of a node being able to rejoin the cluster without manual intervention.
    The approach proposed here is a bit different but valid as well IMO, and probably simpler to implement. And importantly it does not prevent evolving to some more automated form of management in the future.

  3. MariemBaccari commented on Nov 3, 2026

    @MariemBaccari
    ContributorAuthor

    This comment should be addressed as part of this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

raftstoreRelating to the application-layer consensus implementation based on raft

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions