[jll29]

Apache Spark originated from a Berkeley Ph.D. thesis (Zacharia), so it shows that a single person can still move the needle in systems research [1]. It also shows that academic research matters, and that academic innovation interacts with industry and FOSS research: Spark was built as a response to limitations of Hadoop (FOSS research & development), an open source clone of Google Map/Reduce (industry research & development). Spark then led to a startup, Databricks, so we've gone full circle.

Now this doesn't render Pike wrong, his complaint is these things do not happen often enough. My take on why is: many academics are not good coders, because academic environments view coding as "just engineering", all they care is papers. The primacy of the paper (over the system artifact that can be demoed) is one of the sad developments in CS.

Edit: We should less discuss whether Pike is right or wrong - I think he wants to provoke us to prove him wrong by building more radically new systems again and demo them - let's do that!

[1] https://www.kdnuggets.com/2015/05/interview-matei-zaharia-cr...

[kragen]

We did. More than almost anyone, he did.