|
|
|
|
|
|
by mike_heffner
3912 days ago
|
|
|
This was not "indictment" of Storm. We made the analysis and determined that in order to achieve the goals of the business we were not going to be able to take our current system to a 10x scale without a significant investment. Like any complex, distributed computing system there is always the question of when it makes sense to replace the portions of the system that you are not able to reason about sufficiently enough to scale. Superchief is not an off-the-shelf-framework that anyone will be able to drop-in to fulfill their stream processing requirements. We were running on Storm for 2.5 years prior to moving to SC and during that time we truly developed the understanding of the system and our own workload to be able to build SC. SC builds on many assumptions and understandings of our workload that we did not have when we started with Storm. For us we decided this was the path that would lead to the quickest and highest reward, and we were able to justify the effort by not only halving our infrastructure footprint, but being able to scale the next 10x. Other companies, eg. Twitter Heron, have made similar realizations. |
|
|
So, essentially, Storm solved a more generic problem than you needed it to solve. Superchief is a narrower system that more closely aligns with your problem domain, and this let you not only simplify a little but also gain efficiency. Makes sense!
(p.s. the reason I ask is because I am deeply committed to the Apache Storm open source community, as one of the co-authors of streamparse and pystorm.)