[evanxg852000]

(disclaimer: author here) I have haven't used Erlang/Elixir/OTP so I won't be able to answer to your question.

The goal of the framework is to build a data processing pipeline with relatively big tasks, we only have to process tens or hundreds of messages per seconds. Some are CPU intensive and can last 30 seconds. Other tasks are IO intensive.

[strobe]

how much actors is possible to spawn on usual hardware?

[jerf]

Based on the descriptions of the tasks here, they're going to run out of CPU for the actual processing tasks long before they run out of CPU for the actor management. So it doesn't matter whether their actor implementation is capable of spawning 25 million actors doing nothing on commodity hardware when they can probably only keep maybe 2 times or 3 times the number of CPU's worth of actors busy anyhow.

[fulmicoton]

This is indeed correct. Yes, we have comparatively few actors. One thing that is not explained in the blog post however, is that we can run several indexing pipeline on the same machine. We have a bunch of tricks in our framework to run ~1000ish index pipeline on the same server. That's a target of around 10k actors in the most extreme case.

We also send very few messages: the smallest message we send is represent batches of log lines.

As a result, our framework does not really care about performances. On one thread, it can process ~2.5 millions messages on my laptop: this is well above our needs, but it is correct to say that our framework is very slow compared to others.