[jfim]

However, the more concurrency your stack can do, the less this matters. Assuming the probability of a node getting assigned a long request is 1/5, the probability of a node getting assigned 4 of them is (1/5)^4 = 1/625. If your stack can do 20 concurrent requests, it's (1/5)^20 = 1/95367431640625.[1]

Note that this overlooks the fact that a node that has requests queued only drains them at a certain rate, as long requests tend to pile up. However, if your framework is truly concurrent and your nodes are not CPU bound, but rather waiting for other services, it's possible to get much higher concurrency than just 20.

[1] http://www.wolframalpha.com/input/?i=%281%2F5%29%5E20

[philwelch]

Yes, the "just buy more dynos even when better routing would let you get by with less" strategy. I can see the appeal in this, for Heroku.

[jfim]

You misunderstood my reply. It means that each dyno does more work by being concurrent, not buying more dynos.

It's possible to do several hundred requests per second on a single dyno; just because Rails doesn't allow you do to that due to not being concurrent doesn't mean that other stacks can't.

[philwelch]

Rails isn't "not concurrent", it simply uses multiprocess concurrency rather than multithreaded concurrency. Unicorn, for instance, uses a forking model, which is why Heroku recommends it. Ruby 2.0 also favors improves the runtime's copy-on-write performance, which makes it cheaper to fork processes.

Still, there's a resource limit to how much concurrency you can achieve within a dyno as opposed to by buying more dynos. You shouldn't have to scale horizontally just because the routing scheme is inefficient with the width that it has.

If Heroku really wanted to abandon Rails because other web stacks are easier for them to scale with, they should have let us know rather than turning around and shitting on the platform that made them as a business.

[jfim]

Rails is not concurrent. Forking the process does not make it concurrent, either. Parallel, yes, but not concurrent.

Concurrency is the ability to do work on multiple things at once, while parallelism is the ability to execute multiple things at once. For example, an operating system running on a uniprocessor is concurrent, but not parallel. Another example is HAProxy, which is highly concurrent but not parallel(it can handle several thousands connections, but is single-threaded).

The distinction is important when you're trying to scale. Adding more threads/processes does not help, as you quickly reach OS limits(eg: C10k problem). Having a concurrent web stack(nodejs, EventMachine, Play, Xitrum, Twisted, Yaws, etc.) does, as it allows extremely large concurrency with a limited resource impact, whereas adding more processes quickly hits memory limits(at least on Heroku).

[philwelch]

By that definition, it's more typical for databases and load balancers to be highly concurrent. If you have that, then all you need from your app servers is parallelism. Unless you have an especially bad load balancer that does something stupid like distribute requests randomly, at which point your app servers have to do some of their own load balancing.

And even at that point, you're still using more dynos than you would with intelligent routing.

[jfim]

Some web stacks are highly concurrent too (we're running one on Heroku). On a concurrent stack, it doesn't matter if one request takes longer because it does not prevent other requests from completing, so random routing is optimal.

Leastconns routing falls flat in many cases, such as long polling/streaming/websockets, which count as a connection but barely take any resources. In the case where you would have a load balancer that did leastconns, the servers with many long poll requests would end up underloaded(ie. doing nothing) while the ones with few requests would end up serving most of the application.