[jeffreysmith]

Yep, our Spark deployment, like many others, uses YARN and HDFS. EMR has done some great work to make YARN a great target for deployment of jobs using various technologies.

I'm very much not against the Hadoop ecosystem. The ecosystem represents very real progress for data infrastructure. But Hadoop MapReduce is just not what people should be using to build machine learning jobs at scale in 2015.

Spark makes great use of the Hadoop ecosystem, and I'm primarily interested in future innovations in the big data space that try to work with the Hadoop ecosystem instead of trying to supplant it. Modularity and composability benefit us all.