[iagooar]

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications.

If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites.

Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level.

Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions.

I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!

[janekm]

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

[josevalim]

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in.

Distributed Erlang/Elixir has known limitations. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.

The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow is a feature.

And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.

[janekm]

Thanks a lot for the very insightful reply. I'm learning a lot about Erlang from the responses, and a lot of respect for the community as well. Not bad for an on reflection somewhat inflammatory comment :)

[whitepoplar]

This is a great point! What would you say is the best book to really learn OTP?

[RexM]

Designing for Scalability with Erlang/OTP is pretty good so far.

http://shop.oreilly.com/product/0636920024149.do

[Abekkus]

I'd read up on the elixir language itself from the official website, then look into the "Little Elixir and OTP Guidebook"

https://www.manning.com/books/the-little-elixir-and-otp-guid...

[oldpond]

Pragprog has a good selection. https://pragprog.com/titles/category/elixir?f[include_mine]=...

[dtzur]

Shut up and take my money! I'm sold. Thank you for this explanation.

[majidazimi]

What difference does it make? Erlang/OTP distribution doesn't have pluggable architecture. Sooner or later you will reach a point that you have to modify it. Then you are diverging from the original branch which makes it even more difficult to maintain it. You have to merge your additions into every release (minor or major) and test it thoroughly.

A better architecture for a distributed system has a strong composability property. It should be possible to modify every possible aspect of it on a running cluster without introducing downtime.

Write your own standard well documented distribution layer and become independent of underlying technologies.

[josevalim]

What do you mean by architecture here? If you mean the roles different nodes take and their topology, I actually doubt you can be decoupled from your architecture in a distributed system because they directly affect your design and capabilities.

You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.

How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.

And what about process placement? On which side of CAP do you want your registries to sit?

If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.

And most times, it will be good enough.

As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.

[majidazimi]

I did also mention that for fast time to market it is indeed a good idea to use already available tools. By the way writing a distribution layer is not that difficult. Many companies/startups with scalable back-ends that are not using Erlang/OTP have already done it. The point is scalable software requires knowledge and Erlang/OTP is not going to magically solve it. But it seems many fans are trying to promote it such that it is a magic tool that is going to make a shit software become a hyper scalable one. Just look at the comments below that how people have gone crazy.

[jlouis]

Erlang (and by extension Elixir) definitely provides a set of tools which are good for building highly concurrent distributed systems. And your systems are likely to have few errors as well as being resilient, if you know what you are doing.

But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear search is too slow and needs to run faster will be disastrous to your architecture. Likewise, assuming communication is free is equally disastrous.

The problem with building a distribution layer in another language is the effort involved. Most of the companies which pull that off and get stability in addition are usually large, multinational, and has ample amounts of engineering resources to throw at the effort required.

[andy_ppp]

I think,

1) you are misunderstanding the grand parents points. Better to have primitives that scale you to X rather than having to get to X on your own.

2) The grand parent wrote the Elixir language.

3) Making sweeping statements like "A better architecture for a distributed system has a strong composability property" is very easy.

[majidazimi]

> 1) Better to have primitives that scale you to X rather than having to get to X on your own.

For a fast time to market, I do agree with you. But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.

> 2) The grand parent wrote the Elixir language.

I didn't know him, nor I care.

> 3) Making sweeping like "A better architecture for a distributed system has a strong composability property" is very easy.

I'm considering this as an excuse, than a technical argument.

[josevalim]

> But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.

This requires having distributed system experts on your team or having the money to call them from day one, before you even reach the market and before you are even sure you will have custom needs. If you have the need, the expertize and the time, then surely. But writing a distributed system from scratch should certainly not be the first choice (IMO).

[eternalban]

> A better architecture for a distributed system has a strong composability property.

Sure, you want a beautiful lover, but is your object of desire within your reach? One needs to be practical when playing engineer.

Let's try this: Composibility is a highly desired property for distributed systems however correctly scaling composible semantics is a hard problem.

This sort of duality props up all over the place in general engineering, and certainly in software systems engineering.

[adrianN]

Good luck writing something on your own that scales better.

[toast0]

Erlang's core reason for existence is to control telephone switches, which had two independent general purpose computers connected to the physical switch. So reliability, redundancy, recovery, and fault isolation were the core needs; that drove the design for isolated processes with message passing between them. Because Erlang was in the control plane, and only managing the signal path, not passing the signals itself, there wasn't a big need for speed, as long as it wasn't too slow

Fast forward several years, and isolated processes turns out to be a great fit for large SMP systems, and Erlang/beam is now doing signal path work in a lot of places. Erlang tends not to put explicit limits, but some techniques are going to fail at large scale; ex: if you have 50,000 processes across many nodes, sending the same message to each of those processes is going to be slow; sending one message to each node and fanning out from there is going to be faster; in no small part because you've reduced the network bandwidth you're using.

The nice thing when you hit Erlang scaling limits is that almost everything you need to fix is going to be in a pretty simple state. You're not going to find many things that are layers of optimizations on top of hacks on top of optimizations --- they do a good job of keeping things simple, and not optimizing until it's needed (and even then, they usually pick simple optimizations). Keeping things simple goes a really long way (especially with today's enormous servers).

Edited to add: I don't think they've even needed to tweak the vm yet either, just their user space code. That's pretty huge too.

[cdegroot]

That, indeed. When I compare Elixir/Erlang to some other systems I worked on, "shallow" is the word that pops up. You hit a limitation, you dig into some source code, and you find out that it's pretty simple to understand and to fix it. It feels manageable, I've yet have to run into frustrating roadblocks, and that all gives me the confidence that when I do need to scale up, I have a system I will understand and will be able to adapt. It looks like Discord's story confirms that.

[smokeyj]

It sounds like the main benefit to Elixir is that message handling is built into the language. How does that compare to using a message queue service like zeromq?

[toast0]

As macintux, said they don't really compare. Messaging is everywhere in Erlang, in a way that nobody would do with a message queue. For example, you don't read or write to a tcp socket; you receive and send messages to a 'port'. The same is true for file i/o. Rather than calling a method on a shared object, you generally would send a message to a process that owns the state (or a process that manages the state in a database).

Sending messages to processes on other nodes has the same syntax as sending to a process on your node, which makes it easy to run a distributed system. (Ports are different, you'd have to setup a proxy process on the remote node in order to send/receive from that).

Of course, with the base of process to process messaging you can build a higher level messaging queue (see RabbitMQ for a popular message queue built in erlang).

[macintux]

Messaging is implicit in everything Erlang & Elixir do. Bolting on a message queue to software written in another language isn't really comparable (not a value judgement, it's just not really useful to compare them).

[jhgg]

>I don't think they've even needed to tweak the vm yet either, just their user space code.

We haven't really had to. Really only args we use are "+sbt db +zdbbl 32000 +K true" and increasing the default process limit.

[mononcqc]

I have never seen a language or framework used at a large scale that did not require digging, understanding, and tweaking of the underlying machinery for its specific use case.

The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.

[janekm]

That's a good point. The article is certainly a good summary of what's needed to make Erlang/Elixir scale and a reminder that there are no "magic bullets".

[jbhatab]

The solutions they presented were all Elixir + Earlang. It just took some rewrites to get there.

In truly amazing open source fashion, they also made some libraries for other companies to leverage!!! Super big props to Discord for that one. Seriously can't thank their team enough for going above and beyond.

[geogriffin]

I'm an erlang fan but I wouldn't claim message passing is "fast" in the sense that you might be thinking of (though copying data does have some GC and CPU cache performance benefits that are harder to reason about.) There's no magic in Erlang; copying data is copying data. The benefits of Erlang lie elsewhere, and I've heard Joe Armstrong, when asked about BEAM performance, say something along the lines of "why do you care about performance [in this day and age]?"

[_asummers]

With OTP20, copying data isn't always necessary anymore :)

[jlouis]

Copying data is usually necessary in OTP 20 as well I'm afraid. That new optimization doesn't trigger for most of this. But binary data is not copied and hasn't been since at least OTP11 :)

[_asummers]

The release says "Erlang literals". Wouldn't that be things like atoms, integers, booleans, tuples, and the like? That plus binary data should cover a good deal, unless I'm reading too much into the blurb on the release notes.

Under what circumstances does it get triggered (since you seem to be more knowledgable about this than I am!)? I expect records would not fall under this.

[jlouis]

A "literal" in this case is a constant value defined in a module. Those live in a separate space in the VM and are referenced directly because they are immutable and can be shared. If you sent such a literal before OTP20, it would be copied into the heap of the target process. Not anymore.

But it doesn't help with cases where you are constructing a term (dynamically) in a process and sending that term. There is more meat in the blog post of mine: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio...

[lojack]

I read it as "Elixir was a good choice, but it isn't a silver bullet." The fact that they spend most of the article talking about the gotcha's makes their statement about going with Elixir given the opportunity to repeat more favorable in my mind.

[rdtsc]

For a scalable 5M connected users system no language is a silver bullet. Some tools are just better suites for some jobs. Elixir/Erlang OTP just happen to be suited very well for these kind of jobs.

[quaunaut]

I don't think that's what Erlang was made for at all?

Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.

[janekm]

It may well be a misconception on my part - I assumed that as such message passing is such a core feature of using Erlang for distributed systems one should be able to treat it as a low-cost operation.

[quaunaut]

You can generally, just Discord is getting to a level of scale where the issue has more to do with architecture than language. All of their solutions in this post were Elixir-based solutions, and have very clean, easy APIs(and so far not very much code- all those libraries that solved their problems came out to ~400 LOC).

I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."

[sametmax]

Me too. I mean, 5 millions is not that many users. It requires work, but it's not Google size by any mean. It's just a successful service.

Give that the whole selling point of Erlang/Elixir is scalability at the price of the rest, the article is really telling me to avoid the tech.

[passer-by-123]

5 million concurrent users, connected at the same time, sending and receiving data through a persistent connection. Phoenix itself got 2 million on a single node: http://www.phoenixframework.org/blog/the-road-to-2-million-w...

They likely have much more than 5 million users.

To put this amount in perspective, you get 5 million connections after receiving 3000 connections per second, which are never dropped and remain connected, for ~28 minutes. The majority of websites do not get even 300 requests per second.

[tony612]

It's 5M Concurrent Users. And Google-size services can't be built with just a language. It needs much more work to do.

[bsder]

> I see a bright future for Elixir, and a breath of fresh air for Erlang.

Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.

About the only thing I found irritating was a lack of language support for mutable hash tables, but I thought they fixed that at some point.

And Erlang's bit packing/unpacking borders on magical in terms of expressiveness and speed. I wish other languages would adopt it.

[jmcgough]

It became open source in the 90s. It's been used by several high-traffic sites/apps to support chat clients (Riot Games, WhatsApp, and I think Fetlife), but outside of that has seen very little adoption.

I think it's a lot harder to sell people on a language that looks syntactically unintelligible to a lot of web developers. Most people I know working with Elixir now are/were Rubyists, so it's easy for them to make the leap from that and start to understand all of the awesome things they get from BEAM and OTP.

[jhgg]

Elixir/Erlang is great, but in addition to knowing the language, there is a huge library of tools within OTP/ERTS that you need to understand to effectively use it and skills learned from other languages don't map too well to Erlang/Elixir as they would to other languages (like going from Python to Go for example). In addition to learning OTP, you also have to learn how a functional language works if you're not familiar. A lot of things to learn to adopt Elixir/Erlang, but it's well worth it! I just don't think there's much reason for people to make that leap as there are much more familiar languages that can be used to solve the problems most developers are trying to solve.

[quaunaut]

I mean, personally I'm using Elixir more for the language itself than OTP. I have a basic knowledge of OTP now after using the language for 2 years, but FP was my main reason for coming and staying, which for me has been the real fun of it.

Then again, FP for me was not difficult at all, and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition, disappear overnight.

[pmarreck]

> and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition, disappear overnight.

I've noticed that pattern-matching and guards just completely eliminate a stupid* amount of boilerplate logic in Ruby code

* "stupid" in hindsight, of course

[jmcgough]

I fell in love with the language, which is why I kept using it to begin with - working in a functional language is great, and it's very readable and powerful (pattern matching <3).

With that said, I could write basically anything Elixir gives me with Ruby. The real value I get from elixir is the blazingly fast performance of BEAM and the speed at which I can build complex and stable distributed systems with OTP.

[_asummers]

The value to me (other than OTP + friends) is not what Elixir gives you, but what it takes away. Not being able to monkey patch things and change mutable state is really freeing. You can do it if you're diligent, of course (I code Java with as many static methods and final, immutable fields as I can and what I know about Ruby, it has things like freeze), but it's hard to never reach for a tool "just this once" if you're in a time crunch.

[pmarreck]

Syntax is a funny thing. It seems that some developers (perhaps most) care about it (disclaimer: myself included) but there's a large number I've encountered who do not. It seems to be a subjective preferential thing, because most rational discussions I've attempted to have about it result in circling around a drain and coming to no conclusion. I'd say that they're simply blind to it (similar to how a colorblind person can't tell when their clothing color choices just "look wrong"), but they could (perhaps justifiably) argue that they're simply agnostic to it. It's just an aspect of a language that they don't care about and which doesn't influence whether they wish to work with the language or not.

... I'm not like that. I took one look at Elixir and said "HOLY CRAP IT'S A FUNCTIONAL SCALEABLE RUBY!" (of course that was merely scratching the surface, but...)

[eggy]

I tried translating some of the book "The Handbook of Neuroevolution Through Erlang" into LFE (Lisp Flavored Erlang), and Elxir (somebody's done this already I believe).

Robert Virding, a co-creator of Erlang, creator of LFE, also wrote Luerl - Lua implemented in Erlang.

So if syntax is important you have many choices to use the BEAM VM/OTP in a Ruby-like language (Elixr), a Lisp (LFE) or in Lua (Luerl)!

I am now looking at Robert's videos of a game whose logic is written Lua, but it runs an Erlang process for each ship of thousands he spawns in the demo. Pretty cool. There's even Torchcraft - Luerl and Torch ML for learning AI/ML in Starcraft.

I still prefer LFE because I like Lisp syntax, biased as yourself about such things, however, I still think Elixir macros are not as far reaching as LFE's since they are handled after parsing in Elixir vs. LFE [1], although I wouldn't mind being proved incorrect on this.

  [1]  https://groups.google.com/forum/#!topic/lisp-flavoured-erlang/ensAkzcEDQQ

[pmarreck]

How far did you get with the "Neuroevolution Through Erlang" book? I happen to own it (there must be like 10 of us!) but haven't gotten around to it yet...

[eggy]

I am more than a third of the way through. I got distracted, and need to pick it up again. I like the writing style, and that if focuses on TWEANNs and practical examples rather than broad ML. I like the BEAM/OTP but Erlang syntax rubs me the wrong way. I've tried Elixir and LFE, and now I might try Luerl. Not enough time in the day...

[OvermindDL1]

For note, luerl is more of a library for erlang/elixir/BEAM and not a full language, as it is basically just lua's interpreter rewritten in erlang with some modifications to make it fit the system a bit better than normal Lua (there is normal Lua available via a Port though).

[bradmwalker]

Syntax (and to some extent, expressiveness/conciseness) is superficial. How the code feels under change matters more.

[sho]

> Syntax (and to some extent, expressiveness/conciseness) is superficial

I think this kind of attitude is unhelpful at best, and alienating at worst. There are a ton of "serious" programmers, myself included, who care quite deeply about the enjoyableness of the tools we use, and aesthetic/expressive qualities absolutely come into that.

I believe the recent popularity of elixir kind of proves the case. There are many improvements to the package managers & tooling, etc, but the most obvious is to the syntax - which transforms what previously seemed unapproachable into a genuine option. Dismissing any and all such interest as merely "superficial" seems uncharitable, to say the least.

[pmarreck]

I feel they both matter (and more specifically I don't think a well-thought-out syntax is merely superficial), but as this is just a feeling with an anecdotal datapoint of quantity 1, and there being a dearth of evidence for or against "good" syntax (highly subjective, of course)... we're probably at an impasse lol

[di4na]

The problem is not and has never been the syntax. It is the tooling. No build tool. No way to generate a new project. Do it yourself docs. No templating system. Macros ala C. No package manager. Building releases was a dark art.

It is now getting better. But it was a really steep curve to adoption.

[sho]

> The problem is not and has never been the syntax

Call me unserious or shallow or whatever, but for me - it was a tiny bit about the syntax. More than a tiny bit, actually. The other things you mention, all valid, were just additional barriers to something already fairly aesthetically unpleasing.

It's improved and improving rapidly, as you say, which is really great to see.

[mononcqc]

Aside from macros, pretty much all of these have been covered by build tools for a few years now.

[di4na]

I know but iirc your own talk on tooling "we had to steal elixir package management to get one"

[vertex-four]

Huh? rebar, and now rebar3, have existed for a long while. rebar 2.0 was released in 2012.

[rdtsc]

> Is Erlang really that much of a barrier?

It's not. Not for me at least. I prefer Erlang. Maybe I am strange like that.

I found initially the core concepts are hard - that is using using processes for concurrency, functional (immutable) data structures, functional patterns like recursion instead of for loops, the library ecosystems those are harder things. Those are the same in Elixir as well.

Erlang the language itself also simple. Think a bit like C and C++ if you're familiar. Erlang is like C, the language spec is small. Elixir has additional features which make it more expressive but also more complicated (macros, pipes). It's a bit like C++ having templates and classes. You can do more and with them, but it's a also a bit more to learn. I am exaggerating as Elixir is a lot more elegant and consistent than C++, I am just using the analogy to illustrate the idea of simplicity vs power.

[vertex-four]

Unfortunately, a lot of people appear to hate Erlang's syntax to the point that it's a barrier to adoption - or Elixir never would've picked up at all. I don't understand it myself, but there we have it.

[fbernier]

I've seen this argument about the syntax countless times and during my first looks at Erlang code I couldn't understand much of it, but after sitting down and deciding to learn it it's an extremely simple syntax. The hard/long thing to fully understand is truly OTP.

[scaryspooky]

I have never had an easier job writing an MPEG-TS parser than when I used Erlang. It was incredibly easy and concise.

[censhin]

The challenge is going to be adoption, plain and simple. I worked two years on an Erlang project, and most of our frustration came from the lack of community to lean on when we needed help understanding best practices. Also, when you find only one library to solve your problem that has been abandoned for at least two years that is never fun.

Few projects actually need the level of scalability that Erlang brings, and the cost of building and running an Erlang project as an enterprise application is non-trivial. People need to understand this if they want Elixir to succeed where Erlang hasn't.

[jlouis]

The key benefit of Erlang isn't the scalability in my experience. It is resilience. Once your system is deployed, it tend to run with 0 fatalities for months on end. We have systems where we've had 0 maintenance days on them for months and even years.

In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even better, you can often postpone errors in a system which are non-fatal. I often wait until I have a couple of bugs in a system and then I'll work on that system for a few days fixing them all. In systems where even the smallest amount of brittleness destroys them, this isn't really an option.

Another experience of mine is that Erlang systems tend to take a bit longer to write compared to cranking out a fast solution in another language. But the payoff is that your Erlang solution tend to have better robustness as load increases and people start using the system.

The community is small however, so you'll need to write more code yourself in-house as a result. For some problem spaces, this will hurt a lot because there are off-the-shelf solutions in other languages. On the other hand, larger projects which are more specialized can benefit from you knowing your own code better.

[lostcolony]

For anyone reading this, I worked in a shop where we built an Erlang system, choosing the language specifically for the high uptimes we needed. We nailed it so well that IT proactively reboots the system just because they don't trust that it's still working properly when it runs without issue for months on end. Done right, Erlang can be insanely resilient.

[censhin]

I agree with what you said. What I was really trying to get across was the need for a real community and wider adoption.

[tmat]

We're an Elixir shop and I wouldn't change it for the world. We're betting on it heavily.

[toolz]

I think the next big push in elixir is tooling around metrics. Once there are hard numbers to market to non-techies I think we'll see a large shift towards elixir.

[manukall]

I've recently launched https://pryin.io, an application performance monitoring tool made for Elixir and Phoenix. It hooks into Phoenix and gives you insights into how long your request / channels take and what Ecto queries are run / how long those take. You can also manually augment pretty much anything else (background jobs, API calls, ...). Plus it keeps track of some important BEAM metrics like memory consumption.

[gregpardo]

Looks cool. Might want to have a demo site running for people to poke around.

[rubyn00bie]

Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc.

Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).

[_asummers]

We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20).

That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).

[fbernier]

Download count is a weak metric for evaluating the quality of a library. I don't think it's a good idea to skip the few minutes it takes to read parts of the documentation and critical parts of the code.

[_asummers]

I agree completely. I never said that it should be used for evaluation or that you shouldn't read the code or docs.

But then the question is... how do you find high quality libraries without something like download count? The issue is signaling good libraries, with poor discovery. I subscribe to many "libraries worth looking at" mailing lists, the awesome lists, etc. but that is still a pretty coarse net. Searching for libraries on GitHub if you don't know the name already or don't really know what you're specifically looking for is an even worse experience.

Combine that with relatively poor fuzzy search on Hex, where even if you DO know what you're looking for, it leads to a very subpar experience. For example, "e-commerce" leads to different results than "commerce". [0][1] These then exclude [2], which is an ecommerce library available on Hex without that keyword.

[0] https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sor...

[1] https://hex.pm/packages?search=commerce&sort=downloads

[2] https://hex.pm/packages/ryal_core

[ericmj]

Search on hex.pm is not great and can definitely be improved, but we are currently at the limit of postgres full text search afaik, unless there are some magic things to tweak. This means we need to switch to other technologies to improve the search which is a bigger task.

But regardless of how good search is it wouldn't find ryal_core when searching for "commerce" if that word or any similar is nowhere in the metadata. The description for the package ryal_core is "The core of Ryal." which doesn't do the search any favours.

[ramchip]

I don't think that's the problem for metrics. Exometer is the go-to Erlang library, but its dependencies are a nightmare (e.g. https://github.com/Feuerlabs/exometer/issues/154 or https://github.com/Feuerlabs/exometer#dependency-management), it's essentially unmaintained, and integrating it with Ecto or Phoenix is roll-your-own. I think there is very much a need for a more modern library that plays nicely with Hex.

[Aqua_Geek]

> I think there is very much a need for a more modern library [...]

Building a solid standalone monitoring/metrics package is apparently next up on the "todo list" for Chris McCord (creator of Phoenix): https://www.youtube.com/watch?v=pfFpIjFOL-I

[_asummers]

We've had success with Elixometer, though the dependency pinning was a pain. All but two are fully on Hex now.

https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468...

[ramchip]

Thanks, I might have to use this soon :)

[cooervo]

Truly the strenght of elixir comes for Erlang. I wonder if Scala is also as good as erlang for creating robust and fail proof distributed systems?

[lod723]

No, because the JVM currently lacks features to truly isolate threads of execution, which is important for both performance and reliability. AFAIK, the JVM folk aren't even talking about addressing this, which is disappointing.

[justinhj]

Scala paired with Akka is definitely good for that, I don't know how it compares performance wise with Elixir, would be interested to know. We're building a game server backend with lots of messaging in Akka.

[tormeh]

Scala/Akka is a lot faster than Erlang, but it also lacks per-actor GC, which means that latency is going to be higher in a Scala/Akka system. Also, the per-actor heaps in Beam means there's no risk of two Erlang actors having a reference to the same memory, whereas that's easy to do in Scala/Akka.

[OvermindDL1]

Actually you'd be surprised at just how fast the BEAM passes messages, especially as it also includes linking, monitoring, cross-process, cross-system, among other features. When creating a multi-thread atomic messaging system in C++ long ago I was able to out-perform the BEAM (though I'd not rank it as significantly out-perform), but once I added failure handling features among others then it never got close to BEAM's speed again in raw message passing.

However, the BEAM is not very fast on executing actual code, it is like Python in that way where it is good to slave out CPU-heavy work; the BEAM is built for async IO, and yes, running a JVM (or C++) system as a node on a BEAM mesh is fantastic for that (or a port, or NIF for small work).

[biokoda]

The point of Erlang is that it makes writing robust distributed systems easy and fast. The language itself is tiny.

Scala on the other hand is not remotely small and easy to understand.

[corobo]

It convinces me that with a selection of good devs you can hugely scale anything.