[pdimitar]

An app I maintained that was doing anywhere from 3 to 50 DB queries per request, 98% of the time was spent in the DB roundtrips. The Elixir code itself took 0.2ms to 3ms.

It's really lightweight. Can't be anywhere near Go or Rust of course but it's provably much faster than Ruby, Python and PHP, and very often outperforms JS on V8.

Most of the overhead I observed in Elixir apps stems either from external network requests (DBs, K/V stores etc.) or CDNs / static assets. Most of the Elixir code is almost invisible in terms of footprint.

[jashmatthews]

> it's provably much faster than Ruby, Python, PHP

This is simply not true. Ruby + Sinatra/Roda/Hanami + Sequel even performs better than Elixir + Phoenix + Ecto in some cases. It's basically dead-even. For example: https://www.techempower.com/benchmarks/#section=data-r16&hw=...

BEAM has the benefit of being a register-based VM vs the stack based VM of CRuby but the immutable semantics of Erlang/Elixir negate a lot of the VM and GC advantages.

[pdimitar]

You should read a bit more on how the various popular benchmarks manage their different code bases and not just quickly believe the graphs. There's a lot of story behind them.

Several languages, Elixir included, are misrepresented due to badly written and non-idiomatic code (caveat: this was true a while ago, I haven't checked lately though). And when its users try and open a PR they are shot down with "We don't allow cheats". No cheats though, just very normal language best practices.

So I disagree with you here. I maintained a ton of Rails and several Sinatra apps and to say they are on the level of Elixir makes me wonder what kinds of hosting budgets we both had access to and were they really so hugely different.

(Sinatra was actually very decent but it being Ruby it doesn't scale well and Erlang/Elixir work much more stably until they start having problems compared to it.)

Again, don't just trust the popular benchmarking suites. You will quickly discover they usually either have an agenda (like the Crystal language creators always showing how quicker it is compared to Elixir, and refuses any Elixir PRs -- seriously no idea what the Elixir community ever did to them) or only pay lip service to representing all languages fairly (like TechEmpower does).

(Edited because I wrote the first version after a nap and it came off sharper than I intended.)

[bhauer]

There is an implication here that we (TechEmpower) have rejected PRs that would make the code in our Elixir tests more idiomatic. And if I understand you correctly, when we did so, we explained that the implementation did not comply with the spirit of our tests. To be clear, we do reject pull requests that we believe are cheating (e.g., caching results or other means to avoid specific workloads called out in the test requirements).

Can you point to which Elixir-related PR [1] we rejected on that basis?

Some of our most recent Elixir commits were contributed by a gentleman named Michał Muskała. For example, here are some he characterized as "minor optimizations" [2]. Since he is a one of 16 people on the Elixir language organization page [3], I take his contributions as representative of idiomatic code or at least what the organization wants to represent as idiomatic. Do you have a reason to disagree?

[1] https://github.com/TechEmpower/FrameworkBenchmarks/search?q=...

[2] https://github.com/TechEmpower/FrameworkBenchmarks/pull/3848

[3] https://github.com/orgs/elixir-lang/people

[pdimitar]

Thanks for chiming in.

Michał Muskała is a core contributor and his opinion weighs a lot so I am really glad you guys are working with him.

I did check out the PR and his changes seem to be on par with what I've seen in other repositories used in TechEmpower's benchmark. Not cheating, just making sure not to use extra baggage that can severely hamper a framework's performance.

As for rejected PRs, I looked but couldn't find what I roughly had at the back of my head. It was a while ago and since I can't find it now I will have to concede that I remember incorrectly. (Ouch.)

As I pointed out, my info was outdated. I see things have improved in the meantime, which is good news!

Thank you for being responsive and transparent. Much appreciated.

[jashmatthews]

There are plenty of other benchmarks where Ruby on YARV and Elixir on BEAM are roughly even. In this example, JRuby is ~2x faster than Elixir + BEAM.

https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac...

[pdimitar]

As for CRuby and JRuby, they are completely different runtimes. Not sure they belong in a comparison. Elixir only has one runtime, let's compare with the defaults.

Additionally, why do you use the scripting mode of Elixir (.exs file extension)? Why not compile it (.ex file) and run it? The scripting mode hampers performance. Can you update the gist with a compiled version?

And finally, comparing raw performance isn't really what we argued about here (at least I wasn't). I thought we talked about I/O and concurrency capped environments which are the conditions in which the usual web / API app lives.

Erlang/Elixir shine on a lot of things. But definitely not on raw crunching power -- that much is true and the Elixir community is realistic about it.

[jashmatthews]

I actually even tried compiling the module with HiPE but it made almost no difference so I didn't update the Gist. I've updated it to reflect the tiny improvement it made.

In terms of web, CRuby 1.9+ simply uses OS threads for concurrency and parallel IO. You will run out of CPU resources before "the runtime won't scale" on modern Ruby.