request PreVote from non existing IPs

[fibis]

We have since some days an interesting issue. We have a cluster with 3 replicas. Since the cluster is going into an unhealthy state, it's scaling down to one replica.

But this one node is keeping being unhealthy with status "QuorumNotReadyWaitATerm". When I take a look into the logs, I see the following error messages:

W20250526 10:17:48.216668   261 node.cpp:1559] node default_group:10.52.1.18:8107:8108 request PreVote from 10.52.0.3:8107:8108 error: [E111]Fail to connect Socket{id=51539607822 addr=10.52.0.3:8107} (0x0x7b3871815500): Connection refused
W20250526 10:17:48.217233   267 node.cpp:1559] node default_group:10.52.1.18:8107:8108 request PreVote from 10.52.11.2:8107:8108 error: [E111]Fail to connect Socket{id=51539607841 addr=10.52.11.2:8107} (0x0x7b3871818480): Connection refused
I20250526 10:17:49.122349   214 raft_server.cpp:683] Term: 6, pending_queue: 0, last_index: 416717, committed: 0, known_applied: 408601, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 34011468
W20250526 10:17:49.122424   214 raft_server.cpp:708] Node with no leader. Resetting peers of size: 1

So this one node tries to connect to two nodes that don't even exist, since we have nothing behind these IPs addresses. Furthermore, the list of nodes that is mounted includes only the one node. So it looks like that these configuration or IPs are cached also after a restart of the pod. Or is there another way to explain this?

[akyriako]

Hi @fibis, yes it seems that internally typesense, raft namely in this case, is reading a cached(?) version of the nodes list (god knows from where and why) and not the refreshed version from the ConfigMap (which is practically a file in each pod so the new values ARE indeed there because the operator reevaluted the contents of this file in every loop, every 1min). This refresh of the ip addresses from the raft side now, happens every 30sec according to typesense documentation. The operator from his side, acting correct as per design, tries to minimize the "damage" radius and downgrades to a single node cluster as it diagnoses a quorum loss. Why now, the single-node cluster is not re-evaluating a new quorum is actually "an interesting issue" but rather unpleasant one. Thanks for letting me know, I will file it as a bug and open an issue #139. It will help if you could share some logs from the manager and the pod.

[fibis]

Hi @akyriako thanks for the detailed explanation. I will take a look into the logs and add it into the issue and also try to find a solution. It is super interesting, since it has started out of nowhere.