[fnord77]

we have a data processing system in production that, for various reasons, will not scale horizontally without major work to the open source framework we're using.

Nobody has any appetite to fix the semi-abandoned framework, so if we want to process additional data from bigger workloads, we really have no choice but to rewrite our system.

Joel's right - there's a lot of pitfalls. We tried this once and it was a failure. Our 2nd attempt is going much better, though.