[toast0]

By epmd do you really mean all of dist?

The common wisdom on dist is drastically wrong. I was at WhatsApp and going to Erlang conferences and people would say don't run dist clusters with more than N hosts (N ~~ 100), and we were running dist clusters with 10N.

You probably want to adjust socket buffers and dist buffers.

WhatsApp did use something they called wandist to provide cross-cluster connections and more specifically to determine which clusters communicated with which other clusters. That's useful if the node counts are high enough and memory per node low enough that socket buffers are significant and you have a lot of nodes that don't communicate with each other; dist wants to make a full mesh, but you might be able to avoid it other ways? Wandist was also useful because pg2:join had scaling challenges because of contention on cluster-wide locks from the global module, but the new pg (originally contributed by WhatsApp) addresses that. Wandist also added regional affinity for the cross-cluster equivalent of process groups, and as mentioned below, reliable messaging layered on top of multiple connections.

Dist doesn't hide any imperfections of your network though. If you routinely have tcp connections that can't make progress, you will have dist reconnects and have to deal with the unknowns there (wandist layered multiple connections and reliable messaging on top of dist, if a connection stopped functioning, messages would be sent on a second connection; at the cost of using more memory). If your network gets throughput limited, you can see net_adm:ping times of tens of minutes, which is exciting too. All that said, you really do want a reliable network when running distributed systems of any flavor, and building mostly reliable networks is possible, but it's a choice. You'll still need to deal with ocassional issues even with a mostly reliable network; WhatsApp used to routinely catch network problems at our hosting provider before they did; and we had to build a lot of blackbox monitoring and diagnostics for their network.

[sph]

I would love to read books or post-mortems of running Erlang/Elixir at scale. Do you have any recommendations? Have you written any memoirs of your experience at Whatsapp anywhere?

[toast0]

It's been a while since I read it, but I think https://erlang-in-anger.com/ addresses some of this subject matter; I'm not sure if it's still current, many things that used to be terrible are now fine ListA -- ListB with big lists used to be a good way to stall your node (to the point we were considering modifying erlang to remove that operator), but now it does something sensible, if not particularly fast. I have not written anything in an organized fashion, but I write way too many comments on here. I haven't seen much of anything from my colleagues; they must be better than I am at keeping their mouths shut, and just getting things done. :)

There's several talks from Erlang conventions; Rick Reed did several overview type presentations, and we had a couple other presenters on different bits and pieces. But, post 2014, we didn't present nearly as much (got kind of bogged down in post-acquisition stuff).