[tptacek]

You're threading, I presume? We do high speed high connection rate work in EventMachine (for instance, to work with order entry for trading exchanges) and have never seen anything like this --- but we religiously avoid Ruby threads.

[ismarc]

Yeah we are, as it was the least-convoluted method for handling the multiplexing (but wanting to maintain the same handler for all of them). Burned entirely too much time assuming that documented ways to handle things were actually rock solid, not rarely used. The first bug where 3 connections could deadlock everything was unrelated to threading. It was related to how it created identifiers for each connection and would create identical identifiers for two different connections so it would think there was data pending for a socket, but the data had already been read, so it tried to do a blocking read when there wasn't data to read. I never fully tracked down the second bug, but it was more likely related to using threading.

One thing we did have was a large majority of the connections were coming from the same IP (but different source port) and a peak connection rate of ~200 connections per second per server and would last a few seconds (so at any given moment, roughly 1000 connections being established or with data in flight). I'd be really interested in hearing what your traffic patterns are roughly like (and hand-wavy what you have event machine doing, like proxying requests, building/returning its own responses, etc.) if you're willing to share at all.