[fndex]

This is coming from someone who likes Elixir. Not much for its distributed systems features, but mostly because of the language design. I keep hearing everyone talk about how Erlang/Elixir gives everything out of the box and you don't need to worry about Queues, RPC or whatever... But in reality, people don't really recommend using Distributed Erlang that much, on most Elixir gigs I worked, they didn't use Distributed Elixir at all, just plain old Kubernetes, Kafka, Redis and GRPC.

[josevalim]

> I keep hearing everyone talk about how Erlang/Elixir gives everything out of the box and you don't need to worry about Queues, RPC or whatever...

Many companies are using Distributed Erlang but not the way you described: they are using it to exchange messages between nodes running the same version of the software. Imagine that you are building a web application, you can directly exchange live messages between nodes without Redis/RabbitMQ [1]. If you are deploying a machine learning model, you can route requests through multiple nodes and GPUs without additional deps [2]. If you want to see which users are connected to your app, you can exchange this information directly between nodes [3].

In other words, there is a subset of distributed problems that Distributed Erlang solves very well out of the box: homogeneous systems working on ephemeral data. And some of the scenarios above are very common.

If you need to communicate between different systems or you need to persist data, then I 100% agree with you, I would not use Distributed Erlang (at best it would be one part of the solution). I think this distinction gets lost in many conversations and sometimes it leads to false dichotomies: "why Erlang when I have k8s/grpc/redis?" while in practice there is not a lot of overlap. I have written about Erlang/Elixir vs Redis [4] and vs k8s [5] for those interested.

[1]: https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html [2]: https://news.livebook.dev/distributed2-machine-learning-note... [3]: https://hexdocs.pm/phoenix/Phoenix.Presence.html

[4]: https://dashbit.co/blog/you-may-not-need-redis-with-elixir [5]: https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...

[giovannibonetti]

> In other words, there is a subset of distributed problems that Distributed Erlang solves very well out of the box: homogeneous systems working on ephemeral data. And some of the scenarios above are very common.

Speaking of which, I'm looking forward to using Broadway [1] in a new project here in my company. Here, people are using an enterprise integration engine specialized in the healthcare space [2], with built-in single-branch version control and all actions going through the UI.

As I come from a background of several years with Ruby on Rails, I really hope to convince people to use this great library/framework your company created, since RoR is severely lacking when handling heavy concurrency like when gluing multiple APIs in complex workflows. Software engineers are going to love it, but integration analysts are used to IDEs with GUIs, so we'll need to create a pretty admin dashboard to convince them to switch.

[1] https://elixir-broadway.org/ [2] https://rhapsody.health/solutions/rhapsody/

[darkmarmot]

We use it across multiple versions of our software running in the same cluster. As long as you dark launch API changes, it's not much of an issue.

https://www.youtube.com/watch?v=pQ0CvjAJXz4

(Doh, just realized I was replying to Mr. Elixir himself! And you're familiar with our project anyway :)

[weatherlight]

Well, in big orgs, that adopt Elixir/Erlang, along with other technologies with poor concurrency stories, those other ecosystems still need Kubernetes, Kafka, Redis and GRPC, to get by. elixir isn't going to make ruby or python apps magically concurrent. So that make sense.

However, in orgs that are primarily Elixir shops, I don't see a lot of Kafka or gRPC. (Redis is just useful, its more than just a queue and K8s and Elixir/Erlang compliment each other, btw.)

[fndex]

>those other ecosystems still need Kubernetes, Kafka, Redis and GRPC, to get by

And what makes Elixir not need Kafka, Redis or GRPC?

Instead of Redis, you could use ETS for caching. But once you have 2+ instances of your app, you will need to have a centralized caching mechanism, otherwise, each instance will have its own ETS with its own memory, not sharing anything. Unless you decide to use Distributed Erlang and connect all the nodes of your application, which comes with a lot of trouble. Much easier to just use Redis.

And lets say you have multiple teams, each team has its own service(built with Elixir), and you need to have some async communication between those services. What native Elixir solution would you use instead of Kafka?

Same for GRPC. What's the alternative? Connecting all the nodes and using Erlang message-passing?

[mikhailfranco]

I think Elixir/Erlang + Redis pub-sub + PostgreSQL is the sane minimal subset for distributed and scalable systems.

Just say no to Kafka.

[john_the_writer]

Where kafka comes in though is no-code db following. These are super handy when you want to be informed of changes to a table, but don't know which "micro-service" is changing the data. KafkaConnect is very handy.

Though I will concede, it's a bit of a bazooka for a mosquito, sort of thing.

[whaleofatw2022]

After using NATS I don't think I'll ever want to use redis pubsub again.

[conradfr]

ETS is quite faster than using Redis though.

[karmajunkie]

single node though, so you have to add distribution in some manner. For some situations, whether its the "right" way or not, redis ends up being an easier way to go.

ETA: As another couple of comments pointed out, ETS also dies with the node so you've got to handle that also when rolling it.

ETS is cool, but its not a panacea.

[conradfr]

Nebulex has different adapters although I've only use it on a local node which uses ETS and with transient data so I can't comment on them too much.

https://github.com/cabol/nebulex

[weatherlight]

> Connecting all the nodes and using Erlang message-passing? Most of the time, yes.

[cultofmetatron]

well my startup uses distributed elixir. we use it horde to distribute certain long lived processes accross our cluster. that does not exclusive to kuernetes. we user kubernetes to manage and handle vm crashes (its only happened twice ever) as well as porvide consistent network toplogy between the nodes.

that said, having the ability to send messages between machines without having to bring in an external dependency like redis is AWESOME from a maintenance perspective. one less thing to worry about and the system just works. our uptime has been pretty amazing given our small team.

[fndex]

Could you explain a bit more how you are using it?

[cultofmetatron]

our payment processor requires a persistent connection to their server for each of our clients. we use horde to make sure there's only one process responsible for each paying user accross our cluster.

[Joel_Mckay]

Elixir/Erlang often simply replaces most of what Kafka, Redis and GRPC offer.

Also, have a look at the Phoenix Framework Channels examples, as it essentially replaces most simple micro-services architectures.

This recipe book covers the common design examples:

"Designing Elixir Systems with OTP: Write Highly Scalable, Self-Healing Software with Layers" (James Edward Gray, II, Bruce A. Tate)

One day, you too may get annoyed with IT complexity, and articles mostly written by LLM chatbots.

Happy computing, =)

[dangets]

Sasa Juric makes this point in 'Elixir In Action' and some of his talks, where in other languages you need to pull in these other technologies near the start, whereas in Elixir you can use the built-in primitives until you start running into scaling problems.

- https://youtu.be/JvBT4XBdoUE?si=Xo0QXgVSI2HCg8pj&t=2198

[birracerveza]

"just plain old kubernetes" is an oxymoron, at best.

[andsoitis]

> they didn't use Distributed Elixir at all, just plain old Kubernetes, Kafka, Redis and GRPC

There must be a good rationale for that decision. Do you know what it is?

[sodapopcan]

There's a good article about BEAM + k8s by José Valim [0]

[0] https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...

You could certainly get away without some of the other stuff but, as another comment has mentioned, it requires some infra know-how. Like, you can't "just" use ETS as a Redis replacement without setting it up in a way that its data won't get blown away during a blue-green deploy.

[sarchertech]

There may or may not be “good” rationale. Could just be that most people using elixir are coming from other languages/ecosystems where all of that is normal.

Also in my experience, most of the time, the infrastructure team doesn’t know anything about elixir.

[rozap]

Cargo culting happens regardless of language. It's true that fully meshed distribution won't fly for Google or Amazon scale. But 99% of companies will never get to that scale, despite what they'd like to believe. Fully meshed distribution works just fine for many use cases.

[sangnoir]

> It's true that fully meshed distribution won't fly for Google or Amazon scale.

I'm not sure I agree at product level: WhatsApp seems to be scaling rather well. I can't say if their use of Erlang is "fully meshed distribution" or not, but it seems to be flying just fine as the world's number 1 messaging platform.