[legedemon]

Yeah, everyone wants to process their queue as fast as possible but "as fast as possible" practically means a cap on the maximum allowed delay. Otherwise, why stop at 30 workers? Go for 300. 3000?

Also, if the workers shared all the code, you could have used unicorn to fork the processes after the code loading was complete. The 400MB per process would then instantly come down to something ~10MB per process at which point rewriting would have been delayed for another year or so.

[nexneo]

As fast as twilio can accept and process without throttling, beyond that its not much useful.

Unicorn forking benefit is overrated, we used it and we don't see much benefit for long running processes.

Sidekiq is good alternative but that means some rewrite(for our app anyway). Secondly Sidekiq looks mature today, I started working on some of these changes 2 years ago.

[boundlessdreamz]

Can you explain why using sidekiq involve a rewrite? AFAIK, using sidekiq you just have to make sure that your jobs are threadsafe not the whole app which is not very hard.

2 years ago, ruby was not COW friendly. So yeah, there was not much benefit to forking if you were using 1.9.3. Not sure how well does ruby 2.x fares in that respect

[nexneo]

You have to make all code threadsafe that execute from Job or you have to decouple Job code and App code.(which probably be required anyway because I'm not sure sidekiq supports old rubies)

So, in any case you have to rewrite as much as code that I rewrote in Go and decoupled from main App. (its not lot of code, I mentioned in talk)

[boundlessdreamz]

How is unicorn forking relevant in this context? Since they had memory usage problems with workers I assumed they were using resque(which uses forking)/delayed job