Hacker News new | ask | show | jobs
by pixelmonkey 3912 days ago
How much of your decision to replace Storm stemmed from "this would be a fun technical challenge" that you economically justified by cutting server costs? It's hard to tell from the blog post whether you view SuperChief as an "indictment" of running Storm at scale, or whether you just thought building your own stream processor would be a great way to learn new things and cut server costs at the same time.
1 comments
mike_heffner 3912 days ago
This was not "indictment" of Storm. We made the analysis and determined that in order to achieve the goals of the business we were not going to be able to take our current system to a 10x scale without a significant investment. Like any complex, distributed computing system there is always the question of when it makes sense to replace the portions of the system that you are not able to reason about sufficiently enough to scale.

Superchief is not an off-the-shelf-framework that anyone will be able to drop-in to fulfill their stream processing requirements. We were running on Storm for 2.5 years prior to moving to SC and during that time we truly developed the understanding of the system and our own workload to be able to build SC. SC builds on many assumptions and understandings of our workload that we did not have when we started with Storm.

For us we decided this was the path that would lead to the quickest and highest reward, and we were able to justify the effort by not only halving our infrastructure footprint, but being able to scale the next 10x. Other companies, eg. Twitter Heron, have made similar realizations.

link
pixelmonkey 3912 days ago
Makes total sense, especially the part "Superchief is not an off-the-shelf-framework that anyone will be able to drop-in to fulfill their stream processing requirements".

So, essentially, Storm solved a more generic problem than you needed it to solve. Superchief is a narrower system that more closely aligns with your problem domain, and this let you not only simplify a little but also gain efficiency. Makes sense!

(p.s. the reason I ask is because I am deeply committed to the Apache Storm open source community, as one of the co-authors of streamparse and pystorm.)

link