[dzello]

Great post! Makes a lot of sense to move I/O to a thread pool, good thinking :) Now you can control max topology I/O+concurrency not just with Storm but also with thread pool size, yeah? Another question - do you monitor how long a bolt blocks each time waiting for a thread from the pool to be given to it? If the pool has available threads it's instant but if the pool is saturated (backed up on an I/O slowdown for example) it would be good to know?

[manum]

Hi, I'm the author of that post. If a thread is not available from the pool the calling thread tries to execute the operation itself. That allows the bolt to continue making progress but adds a simple back pressure mechanism.

This also avoids the overhead of any sort of locking to wait for a thread to be free. Since the number of bolt executors will always be much less than the number of threads in the IO pool it doesn't add too much the concurrency overall. Hope that makes sense.