[mgachka]

Hi,

It's a good read. I have a few questions/comments:

- given the description of the Rumor-mongering approach vs the Anti-entropy approach, it looks like the Anti-entropy approach:

-- has an important overhead in terms of network/messages sent (since nodes are always chatting even when there are no changes in the cluster).

-- is slower to propagate a change of cluster state to all the nodes. Does it mean that in case of a node failures/shutdown, the cluster will be instable for longer (since dead nodes will receive queries)?

- the article mention a "seed node" but doesn't define what this is.

- on a dynamic quickwit setup (that often downscales/upscales depending on the load), it seems that the size of the metadata in each node will keep increasing since the state of the dead nodes will be kept, unless the same node_unique_id can be reused after a downscale/upscale (but I don't know enough how kube works to see if it's the case).

[fulmicoton]

This is correct yes. Rumor-mongering is more efficient both in term of detecting/propagating a node failure and in term of network overhead.

We stopped using SWIM because it is too hard to get right, and because scuttlebutt allows all nodes to expose a bunch of meta-information about themselves.

It took a lot of time and effort to Hashicorp to patch memberlist into what it is today.

[evanxg852000]

- In term of message overhead there is not much difference. one details is that each node keep incrementing his heartbeat counter continuously, making it a state change that need to be propagated.

- slower here depends on the number of node selected to gossip with. if you choose a long gossip period yes, but in real system the gossip period is short enough to not notice it. So slower means compared to Rumor-mongering style. Not as in having an inconsistent cluster state.

-(Thanks) I will probably amend this as a note: A seed node can be any node in the cluster we know is reliable and will almost always be there; kind of like the ACE within your cluster, they can be many.

-Garbage collection :"... We mitigate this by periodically garbage collecting obsolete nodes."