[Jare]

> Node handles one request at a time. It isn't multithreaded. It will receive a request, process that request and return a response. If another request comes in at the same time another request is in process, it is queued until the currently processing request is finished.

I'm missing something here. Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiting for async operations: database, external APIs or other types of I/O usually. That's the very core idea of evented servers.

So, my model is that i.e. if a node process receives 100 requests over the period of 1 second, and each request takes 3 seconds to process but most of that time is spent waiting for async, then the 100 responses will be sent back essentially 3 seconds after they arrived, no queuing to speak of.

From your description, routers do not send multiple requests to the same dyno even if dynos could handle them, and only have a limited amount of dynos they talk to. So queuing is happening in the routers, while dynos idle away waiting for async.

This would be complementary to the problem described by Rapgenius, and mean that the Heroku architecture does not play well with any type of server, neither evented (Node, yours) nor sequential (Rails, as shown by Rapgenius) nor presumably multithreaded or multiprocess (which effectively behaves like evented to the outside world). A huge mess indeed!

[latchkey]

> Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiting for async operations: database, external APIs or other types of I/O usually. That's the very core idea of evented servers.

Within a single request, Node can async its dealing with outside services (databases, api's, etc), but it is still only processing one request at a time. There is no 'synchronized' keyword in javascript. ;-)

Check this out:

https://devcenter.heroku.com/articles/http-routing#heroku-he...

There is an interesting header in there: X-Heroku-Dynos-In-Use. From what I understand, this header is the number of dynos that a router is communicating with. For us, this is always around 2-3.

I suspect that the router is just a dumb nginx process sitting in front of my app. It is setup to communicate with 2-3 of my dyno's in a round robin fashion. If any one of those dyno's doesn't process the request fast enough, then requests start to back up. Once requests start to back up past 30s worth of execution, the router starts just killing those queued requests instead of just leaving them in a queue or sending the requests to another set of dyno's. Even worse is if you have a dyno that crashes (nodejs likes to crash at the first sign of an exception). I suspect that is why we see 2 or 3 in that header.

I think that part of the problem is that the routers don't just start talking to more dyno's if you have them available. So, it doesn't matter if you have 50, 200, 500 dyno's because the router is always only talking to a small subset of them. Even if you add more dyno's in the middle of heavy requests, you are still stuck with H12 for the existing dyno's. A full system restart is necessary then.

[Jare]

By 'processing' I mean the Node application has received the request and has not yet sent the response, i.e. the connection is still alive. 'synchronized' has no bearing here. If the request processing is purely CPU-bound with no async operations then only one request will be processed at any time, otherwise Node will happily process up to thousands of requests simultaneously. This is the ideal use case for Node. It should be trivial to log the amount of simultaneous requests being processed.

According to Heroku docs, Cedar routers do not do any queuing and just serve requests immediately to any random dyno. They are pretty clear on this in multiple places, specifically talking about concurrent requests in Node. They also mention a 'routing mesh', which suggests there are many routers doing their thing. But that header you see maybe not be relevant to Cedar, just like the other header 'X-Heroku-Queue-Depth' should not apply to Cedar either.

[hpguy]

> Within a single request, Node can async its dealing with outside services (databases, api's, etc), but it is still only processing one request at a time. There is no 'synchronized' keyword in javascript. ;-)

This is as wrong as it can get. Multiple requests are processed "concurrently", in the sense that a request gets served as soon as an existing request awaits on async calls. It is different from thread-based concurrency and thus there is no need for things like "synchronized" keyword.

[latchkey]

I think we are saying the same thing. When I say 'it is still only processing one request at a time', I'm referring to my code execution, not the async stuff or anything else within Node.

I found this to be a good explanation...

http://blog.mixu.net/2011/02/01/understanding-the-node-js-ev...