[jeffreysmith]

I think that the Hadoop ecosystem has just expanded. Spark is now a key part of that ecosystem.

Hadoop's MapReduce implementation is arguably quickly becoming obsolete with so many more powerful platforms beating it on several dimensions (not just Spark).

That said, if your interest is machine learning specifically, I can't sell Spark hard enough. Functional programming is so critical for modern, large-scale machine learning. Spark is an absolute revelation for the machine learning developer. It's not the only stack; the PyData stack is totally worthy of study and use. But for big data machine learning, Spark is as perfect as anything has ever been.

If you're really interested in learning more about how Spark, Scala, and functional programming come together in a machine learning system, I'm writing a book on reactive machine learning: http://www.reactivemachinelearning.com/ In it, I'm trying to cover the how and why of different tools, with a focus on Scala, Spark, and Akka.