[rgblambda]

That strong following might not last. My company made the decision to have PySpark be the default for training and future greenfield Spark projects, and they're only really making official what's been the reality for the past few years. A shame, as Scala seems like a nice language but I've never really gotten into it as Python is always the preferred option.

[michaelmior]

It's true that PySpark probably is overtaking (or already has overtaken) Spark with Scala. I still personally like Scala over Java in general for non-Spark projects as it's much less verbose which is my main problem with Java. However, I also haven't used all the new Java features to see how that would adjust the comparison for me.