[mark_l_watson]

I have also found Spark (and Hadoop before that) a little clunky to prototype and develop on, but when you need to handle very large data sets with good throughout performance then systems like Spark/Hadoop are great. One problem they had was maintaining infrastructure, and to be honest, when I used mapreduce as a contractor at Google or AWS Elastic MapReduce as a consultant I didn’t have to deal too much with infrastructure.

Anyway, it makes sense that they backed off using Spark and HDFS - makes sense given the size of their datasets.

The original poster mentioned that their data analytics software is written in Haskell. I would like to see a write up on that.

EDIT: I see that they do have two articles on their blog on their use of Haskell.