[renaudg]

It looks like they have built an interesting, robust and scalable system which is perfectly tailored to their needs.

If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like https://socketcluster.io doesn't provide ?

[lostcolony]

Yes. Discord is served over HTTPS. :P (your link is broken; socketcluster.io doesn't serve over HTTPS)

But seriously, Discord actually benchmarked 5 million concurrent users, horizontally distributed, and having to ferry messages across the cluster, with specifically tailored fanout patterns (rather than just a global pub/sub. I.e., who a message goes to varies, rather than just "everyone").

Socketcluster.io only has benchmarks for a single machine, capped at 42k concurrent connections (though to be fair that was due to them running a single client, rather than a limitation of the server). They don't out of the box support horizontal scaling; you're required to spin up your own message queue solution for that.

So, basically, you're advocating a technology that solves -the simplest part of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any of the custom tweaking Discord describes, solve that AND more of the actual problem Discord had. So...yes, and no. Yes, there is plenty here they've describe that is not available in socketcluster.io, but no, nothing they've done here is no generally solved by an off the shelf system, because they're -using- an off the shelf system, Elixir + Phoenix.

[jondubois]

Hi, main author of SocketCluster here.

SC does support automatic horizontal scaling across any number of machines out of the box if you're running it on Kubernetes.

There's also a CLI tool to deploy it automatically to any Kubernetes cluster: https://www.npmjs.com/package/baasil

See https://github.com/SocketCluster/socketcluster/blob/master/s...

[lostcolony]

I just meant you still need a third party MQ to be spun up (per docs here - http://socketcluster.io/#!/docs/scaling-horizontally). Without that, there is no distribution happening.

From my understanding, you're basically saying "You can combine SocketCluster with the MQ of your choice (the installation and configuration of which is left as an exercise to the reader) and then between Docker, Kubernetes, and Baasil you can orchestrate and deploy it across a cluster". That sounds a bit more complex than just using SocketCluster, which is what the OP seemed to be indicating was all you needed, and is also including the DevOps story, which I don't think either he or I was intending to include.

I was not trying to indicate that SocketCluster can't be -used- to scale websockets horizontally, but that it's not just an off the shelf solution that would have solved Discord's problem either. It requires other parts, as both the docs and you mention.

I'll also reiterate from my post, SocketCluster has no benchmarks pertaining to what happens when you -do- scale horizontally (per docs here - http://socketcluster.io/#!/performance ). That lack alone would kill my interest in it (as would scc-state being a single instance, which would make fault tolerance a real concern to me, but it looks like you know that already). Is performing horizontal scalability tests on the roadmap?

[jondubois]

If you use SCC, then you don't need a separate MQ - That is only if you want to do things yourself manually. I will update the docs to make that clearer.

It should only take a few minutes to deploy a cluster across hundreds of machines. The only limit is the maximum number of hosts that Kubernetes itself can handle (which is I think is over 1000 now)? SCC is self-sharding and runs and scales itself automatically with no downtime.

You can easily handle 5 million concurrent users with a small cluster. SC's problem isn't scalability, it's marketing.

[lostcolony]

That's perfectly fair; fix the marketing then. :P In evaluating a solution, the marketing is the -first thing- anyone looks at. And how it currently reads, "SocketCluster only provides you the interface for synchronizing instance channels - It doesn't care what technology/implementation you use behind the scenes to make this work" definitely reads as "You need a technology/implementation behind the scene" rather than "we provide you a default one, and you can swap it out".

For me to pick Socketcluster for a distributed solution (or more broadly, what I'd want for -any- technical solution) I'd want to know what else I need to pair it with (which the docs actually mislead me on), what else I can benefit from (which the docs don't tell me, but which does exist per your links), and what benefits I stand to get from using it (the docs tell me only marketing claims, but with no metrics, performance, data, etc, for what happens in a distributed context, well, I would avoid it).

Ideally, set up a clustered performance test, and then make as many of the artifacts (docker images, configs, readme, etc) available so others can conduct the same performance test themselves (as well as have a reference architecture for their own solution). Heck, if you're doing it in AWS, consider making the AMIs available along with whatever modifications need to happen. -That- would be very convincing for someone looking to adopt a solution in this space, if they could literally just spin up some EC2s and immediately start throwing load at a fully configured cluster.

Also, to make it clear, is this handling message passing between instances in the cluster?

[jondubois]

Thanks for the advice.

Yes, it handles message passing between instances in the cluster. That means if you publish a message on a channel whilst connected to one host, the message will also reach subscribers to that channel which are on any other host in the cluster. It shards all channels across available brokers, when you scale up the number of brokers, it will automatically migrate the shards across available brokers with no downtime.