I've had this conversation on HN three times now, and I'm misunderstood each time I say it. It's getting a bit frustrating.
I'm not saying that the Erlang maintainers haven't been trying to work on perf. I'm saying that their approach to working on perf has, until recently, been to focus on building on top of large blobs of relatively-opaque third-party code — either in-tree code, like HiPE; or library code, like LLVM. These approaches haven't been maintainable, and have eventually been dropped.
The difference in the last 1.5 years is that the perf enhancement this time is purely due to optimizations to the emulator and runtime — and improvements to the way the compiler works enabling better runtime insight into the code — with no big blobs of opaque code being relied upon. There's nothing extra to maintain; no big experimental perf-hack to enable with a feature flag. It's just the runtime itself being improved.
With previous attempts, the perf improvement was a sigmoid sawtooth: perf increased, levelled off, and then dropped back down. This time, the perf improvement is a linear ratchet function.
---
My other point, is that until fairly recently, most of Erlang's biggest users were "enterprise" customers with embedded use-cases, whose #1 concern was stability — they didn't care about perf-enhancing new language features, because they weren't about to rewrite their working+tested+certified code just to get better perf. They were only interested in "transparent" perf enhancements.
More recently, though, large non-"enterprise" corporate users of the Erlang runtime, like WhatsApp and Discord, have been migrating their work developing third-party perf-enhancing feature modules upstream as language features, because they are interested in rewriting code to increase perf — as long as that rewritten code is also just as maintainable (if not moreso.)
> I'm saying that their approach to working on perf has, until recently, been to focus on building on top of large blobs
Totally agree.
Off topic: I really love Erlang/Elixir - but I'm beginning to losing hope in being able to see a path where it can 3x improve performance. There has be a lot of people for a very long time been hoping this major improvement will come "soon". I'm unfortunately just not seeing it happening or a path to it ever happening. I hope I'm wrong.
I'm not saying that the Erlang maintainers haven't been trying to work on perf. I'm saying that their approach to working on perf has, until recently, been to focus on building on top of large blobs of relatively-opaque third-party code — either in-tree code, like HiPE; or library code, like LLVM. These approaches haven't been maintainable, and have eventually been dropped.
The difference in the last 1.5 years is that the perf enhancement this time is purely due to optimizations to the emulator and runtime — and improvements to the way the compiler works enabling better runtime insight into the code — with no big blobs of opaque code being relied upon. There's nothing extra to maintain; no big experimental perf-hack to enable with a feature flag. It's just the runtime itself being improved.
With previous attempts, the perf improvement was a sigmoid sawtooth: perf increased, levelled off, and then dropped back down. This time, the perf improvement is a linear ratchet function.
---
My other point, is that until fairly recently, most of Erlang's biggest users were "enterprise" customers with embedded use-cases, whose #1 concern was stability — they didn't care about perf-enhancing new language features, because they weren't about to rewrite their working+tested+certified code just to get better perf. They were only interested in "transparent" perf enhancements.
More recently, though, large non-"enterprise" corporate users of the Erlang runtime, like WhatsApp and Discord, have been migrating their work developing third-party perf-enhancing feature modules upstream as language features, because they are interested in rewriting code to increase perf — as long as that rewritten code is also just as maintainable (if not moreso.)