[alberth]

Raw performance is the actual albatross.

IMHO, the low raw performance of Erlang is what's holding it back from mass adoption.

(Even marquee Erlang/Elixir users like Discord, still have to use Rust NIFs to overcome the slow Erlang runtime)

People have a hard time understanding how Erlang can have such: high concurrency, low latency & tight standard deviations ... when people are just accustom to looking at raw performance benchmarks (where Erlang does quite poorly).

While BeamASM (JIT) is very exciting, the reality is that the Erlang runtime has only speed up by ~25% over the last decade, where other languages like JS, Go, PHP have seen >150% speed ups (and Erlang was already considerably slower than these languages prior to their speed ups).

[jacquesm]

Erlang excels at the 'hard' parts of the problem space having to do with complex distributed ultra reliable systems, the 'number crunching' bits are offloaded to specialist languages because the underlying language model that Erlang uses does not lend itself for the kind of optimization that JS, Go, PHP etc have available to them simply because their runtime model is so completely different.

The biggest issue from what I know about how Erlang works internally is not so much the fact that Erlang could not in principle be made fast (and indeed, BeamASM will if it makes it to mainstream become a good step in the right direction), but that the whole way in which Erlang schedules its threads is super ineffecient in terms of cache use and besides the byte code interpreter relies on its ability to keep track of the number of reductions that it has done to determine when a thread has had enough cycles and we need to move on.

This means there will always be a fairly hard upper limit as to how far you can optimize Erlang byte code, it is at its core a cooperative multi tasking operating system inside a user process.

One of the more elegant ways to do such interop is to isolate your number crunching code to a separate process group working their way through a queue one unit of work at the time that way you can use all of the Erlang goodies and still get very good performance.

[derefr]

> One of the more elegant ways to do such interop is to isolate your number crunching code to a separate process group

Or, if you want to maximize perf inside some hot section of code (say, a game-engine renderer) while also retaining Erlang semantics for how it interacts with the rest of the system, you can write your code as a "threaded NIF" — a native thread (in C or Rust or whatever) that sends messages back over to the arbitrary Erlang processes in its address-space (not just its owner process!) as it works.

[alberth]

What concerns me is I see people talk about using NIFs like no big deal.

BUT, they come at the cost of potentially bringing down the entire Erlang runtime. People use Erlang, more times than not, because they want a fail-safe runtime ... so recommending NIFs shouldn't be taken lightly.

Just my 2 cents.

[derefr]

> People use Erlang, more times than not, because they want a fail-safe runtime

I disagree; 99% of people running Erlang runtimes are running them in use-cases where it'd be fine if any individual node crashed and got restarted.

(E.g. running a Phoenix web-app as a K8s replicated Deployment, with no durable internal state, only the mesh-replicated Phoenix.Presence data that can be recovered from peers after node restart; where, for users, even having a long-running websocket is just an optimization over long-polling, and so it's no skin off the clients' backs if said connection gets dropped once in a blue moon, requiring them to reconnect to another node in the cluster.)

I agree that you should only be using NIFs after you've exhausted the other options — "C nodes" being my favorite of those, personally.

But downplaying NIFs ignores a whole other segment of use-cases, where you're not adding native code to Erlang, but rather you're starting with a "core" of native code (e.g. an HFT trading engine), and then building an app by wrapping that native code in Erlang, where the code presents itself to Erlang as a NIF.

If efficient shared-address-space high-throughput interaction between your native "core" and your higher-level "glue" is the whole point, then Erlang+NIFs is a pretty great way to get a bunch of advantages without losing much. It's a lot better than what you get from the alternatives — where those alternatives are "loading the native core as an FFI module in some other dynamic language runtime."

[the_only_law]

> the 'number crunching' bits are offloaded to specialist languages because the underlying language model that Erlang uses does not lend itself for the kind of optimization that JS, Go, PHP etc have available to them simply because their runtime model is so completely different.

Yeah I love Erlang, but often reach for something else when I need to be able to express and optimize compute bound problems.

[jacquesm]

In a way it makes perfect sense. When I look into my toolbox I see a whole pile of different saws, each of which has a specific usecase, metal, wood, living wood, trees, joinery, figure work, glass+ etc. The one-tool-to-fit-all-usecases programming language hasn't been invented yet as far as I can see.

+ technically, probably more of a grinder because the blade is covered with diamond dust, but you use it like a saw.

[MrWiffles]

> The one-tool-to-fit-all-usecases programming language hasn't been invented yet as far as I can see.

Well, just like you can use a screwdriver to pound a nail into the wall (flip it handle-first), you could use C or Rust or Go or Java or...nearly anything to build...nearly anything if you really _forced_ it hard enough, had infinite time, expert labor and budget, but...

...there are better tools for the job of pounding a nail into the wall than a screwdriver! ;-)

In other words, I totally agree with you. Sure, you could build, oh I dunno, say a compute-intensive application entirely in, say, Ruby, but you'd need to throw some ungodly expensive hardware at it. Makes a lot more sense to use the right tool for the job!

[derefr]

> While BeamASM (JIT) is very exciting, the reality is that the Erlang runtime has only speed up by ~25% over the last decade

Yes... because perf hasn't really been a focus for Erlang until about 1.5 years ago. Various (usually academic) third parties have contributed big performance-enhancing patchsets (the earlier JIT; Dialyzer and then HiPE based on Dialyzer analysis; etc.) but these have then languished, with attempts to further development on them slowing to a crawl over time. The patchsets were just "too big to be digested properly"† by a team of core maintainers who 1. didn't write the code, and don't fully understand it, and 2. who aren't CS academics themselves.

Each of these indigestible patchsets was eventually dropped, throwing away any follow-on in-tree work done to it, and dropping perf back to where it was before said patchset was introduced.

The new work is in-tree, done ground-up by the core maintainers themselves, and so is actually showing linear improvements in speed. This not only includes runtime speed, but also the batteries-included addition of long-ignored performance-oriented features, like atomic counters and fast global readonly modules (and the runtime itself being gradually rewritten in terms of these!)

† This is exactly why, in the Linux kernel, big patchsets aren't accepted as-is, but rather are required to be broken down into small changes that 1. can add value on their own, and 2. can be molded to fit the design philosophy of the kernel on their own. This is why e.g. the "containers" patch from OpenVZ was never pulled in; but instead, each bit of it was gradually reworked into the cgroups + namespaces code that powers Linux containers today.

[alberth]

> perf hasn't really been a focus for Erlang until about 1.5 years ago.

The core maintainers have been working on JIT/perf for 10-years.

You can see Lukas own described 10-year journey documented below.

https://drive.google.com/file/d/1hHCs90kDX_wJ9AbLzGNu5bZKdIV...

[derefr]

I've had this conversation on HN three times now, and I'm misunderstood each time I say it. It's getting a bit frustrating.

I'm not saying that the Erlang maintainers haven't been trying to work on perf. I'm saying that their approach to working on perf has, until recently, been to focus on building on top of large blobs of relatively-opaque third-party code — either in-tree code, like HiPE; or library code, like LLVM. These approaches haven't been maintainable, and have eventually been dropped.

The difference in the last 1.5 years is that the perf enhancement this time is purely due to optimizations to the emulator and runtime — and improvements to the way the compiler works enabling better runtime insight into the code — with no big blobs of opaque code being relied upon. There's nothing extra to maintain; no big experimental perf-hack to enable with a feature flag. It's just the runtime itself being improved.

With previous attempts, the perf improvement was a sigmoid sawtooth: perf increased, levelled off, and then dropped back down. This time, the perf improvement is a linear ratchet function.

---

My other point, is that until fairly recently, most of Erlang's biggest users were "enterprise" customers with embedded use-cases, whose #1 concern was stability — they didn't care about perf-enhancing new language features, because they weren't about to rewrite their working+tested+certified code just to get better perf. They were only interested in "transparent" perf enhancements.

More recently, though, large non-"enterprise" corporate users of the Erlang runtime, like WhatsApp and Discord, have been migrating their work developing third-party perf-enhancing feature modules upstream as language features, because they are interested in rewriting code to increase perf — as long as that rewritten code is also just as maintainable (if not moreso.)

[alberth]

> I'm saying that their approach to working on perf has, until recently, been to focus on building on top of large blobs

Totally agree.

Off topic: I really love Erlang/Elixir - but I'm beginning to losing hope in being able to see a path where it can 3x improve performance. There has be a lot of people for a very long time been hoping this major improvement will come "soon". I'm unfortunately just not seeing it happening or a path to it ever happening. I hope I'm wrong.

[MrWiffles]

As an Elixir noob, I'm reading these comments about performance and coming to some conclusions that I'd like to ask ya'll to idiot-check me on.

First, let me establish that there are at least two ways of thinking about performance in the context of a server-side HTTP application:

1) Per-request speed ("from receiving the request to response, single-threaded, X ms") 2) Throughput-based metrics ("We can serve X requests per second with Y cores spread across Z nodes")

I think both ways of measurement have value, but in this context I'm operating under the assumption that we're focused on option 2. Correct me if I'm wrong.

Now, with that understanding in place, it sounds to me like Elixir/Erlang performs "pretty good" in terms of that overall throughput if you leverage the multithreading capabilities in building your apps and thread scheduling isn't terribly contentious (hopefully not a lot of other processes/threads competing for execution time on the same machine as your Elixir app), but measured on a raw per-operation speed metric it doesn't quite stack up to other non-compiled languages like PHP or maybe even Ruby (which would shock me but let's just poke that bear anyway, see what kinda growl we get...).

So if indeed my understanding is right, my next question is: "how much of a performance reduction would we realistically see with an Elixir app that's otherwise well built and doing the same thing?" Issues of misconfiguration, bad deployment architecture, etc. withstanding.

And if that question is indeed valid/not-batshit-crazy, my next would be: "does that degredation matter or scale in a multi-node/clustered deployment context?" In other words, if you have a say 50-node (be they VMs or containers) Elixir app, built well, properly configured and deployed, load balanced etc., is it going to have significantly/noticeably slower throughput than an equivalent PHP/Java/Ruby/$OTHERLANG app when compared as close to apples-to-apples as you can get that sort of thing?

(Maybe take Ruby out of the equation here with issues of GIL/threading being what they are over there. Haven't had the chance to work with recent Ruby though so if threading is a valid comparison nowadays, by all means keep it in mind!)

I'm just trying to get a sense for whether or not I'm barkin' up the wrong tree with my recent interest in Elixir. Am I wasting my time right now and maybe should wait a few years before really tearing into it, or is it effectively in a real-world, non-academic scenario, going to perform "close enough" in the real world in a multi-core, multi-node load balanced context?

[Communitivity]

I am dubious it's the performance of Erlang that's the albatross, considering the performance of RabbitMQ messaging system and Process One's XMPP server ejabberd. However, at the end of the day Erlang is a VM language, so I'd think it's not going to be as fast in many cases as Rust. Performance in a distributed system is good, because of the design, but that's still not the #1 priority of Erlang. I think Erlang's #1 priority is robustness.

Then what is the albatross?

I think it is complexity. Erlang, and OTP, is not simple to understand. There are a lot of footguns. It is beautifully designed and fit for purpose, given it's history and heritage. However, the learning time for creating a real app in Erlang is high. To flip it around, I think the reason Ruby caught on so fast was that it was so easy to learn Rails and create a web app with it.

You could perhaps convert Elixir to be based on top of Rust, but there are many concepts that don't map, or don't map well.

[conradfr]

FWIW I have ported Json APIs from JS & PHP to Elixir and the Elixir ones were the fastest (on the same server). And it was before even leveraging ETS to cache DB data (which is faster than Redis by design).