[l72]

I would be very curious to see how well this actually works in different data centers. My understanding of empd is that it basically expects all nodes will be on the same switch. I don't believe there is any default handling for spikes in latency or netsplits, and epmd is known to be pretty chatty.

The common wisdom seems to be to do federation of clusters rather than clustering multiple data centers together.

[toast0]

By epmd do you really mean all of dist?

The common wisdom on dist is drastically wrong. I was at WhatsApp and going to Erlang conferences and people would say don't run dist clusters with more than N hosts (N ~~ 100), and we were running dist clusters with 10N.

You probably want to adjust socket buffers and dist buffers.

WhatsApp did use something they called wandist to provide cross-cluster connections and more specifically to determine which clusters communicated with which other clusters. That's useful if the node counts are high enough and memory per node low enough that socket buffers are significant and you have a lot of nodes that don't communicate with each other; dist wants to make a full mesh, but you might be able to avoid it other ways? Wandist was also useful because pg2:join had scaling challenges because of contention on cluster-wide locks from the global module, but the new pg (originally contributed by WhatsApp) addresses that. Wandist also added regional affinity for the cross-cluster equivalent of process groups, and as mentioned below, reliable messaging layered on top of multiple connections.

Dist doesn't hide any imperfections of your network though. If you routinely have tcp connections that can't make progress, you will have dist reconnects and have to deal with the unknowns there (wandist layered multiple connections and reliable messaging on top of dist, if a connection stopped functioning, messages would be sent on a second connection; at the cost of using more memory). If your network gets throughput limited, you can see net_adm:ping times of tens of minutes, which is exciting too. All that said, you really do want a reliable network when running distributed systems of any flavor, and building mostly reliable networks is possible, but it's a choice. You'll still need to deal with ocassional issues even with a mostly reliable network; WhatsApp used to routinely catch network problems at our hosting provider before they did; and we had to build a lot of blackbox monitoring and diagnostics for their network.

[sph]

I would love to read books or post-mortems of running Erlang/Elixir at scale. Do you have any recommendations? Have you written any memoirs of your experience at Whatsapp anywhere?

[toast0]

It's been a while since I read it, but I think https://erlang-in-anger.com/ addresses some of this subject matter; I'm not sure if it's still current, many things that used to be terrible are now fine ListA -- ListB with big lists used to be a good way to stall your node (to the point we were considering modifying erlang to remove that operator), but now it does something sensible, if not particularly fast. I have not written anything in an organized fashion, but I write way too many comments on here. I haven't seen much of anything from my colleagues; they must be better than I am at keeping their mouths shut, and just getting things done. :)

There's several talks from Erlang conventions; Rick Reed did several overview type presentations, and we had a couple other presenters on different bits and pieces. But, post 2014, we didn't present nearly as much (got kind of bogged down in post-acquisition stuff).

[victorbjorklund]

Doesnt seem to be a problem for all the elixir apps running as clusters on fly.io.

[benmmurphy]

I think there might be some gotchas with some of the Erlang clustering primitives. I worked on a product that Erlang solutions helped design and they didn't use Erlang clustering for RPC but rather reimplemented it themselves over dedicated TCP connections. If you do some googling there are people complaining about issues in Erlang RPC caused by multiplexing the RPC messages between nodes and cluster meta messages over the same TCP connection creating head of line blocking issues. Though, maybe some of this has been fixed since:

https://www.erlang.org/blog/otp-22-highlights/#fragmented-di...

There are also apparently other bottlenecks like funnelling everything through a single gen_server. (https://erlang.org/pipermail/erlang-questions/2016-February/...). Again, this may have been fixed since then.

[bo0tzz]

It's probably not a problem during normal operation, but I'd be worried about what happens if the network misbehaves.