[apohn]

>There's the issue of PySpark, as opposed to Spark itself.

I'm a little confused by this comment.

PySpark is the Python interface you use for Spark. IME, PySpark is actually a really nice API. Your other options are Scala or Java. I think there's a R interface as well, but I'm think that lags behind PySpark.

Saying that PySpark is dying, but Spark is not is a very contradictory thing to say. If you look over the universe of Spark users, I'll bet there are more Python users than Java or Scala. There would have to be a very big shift in the Spark userbase for people to decide that PySpark is going to be deprecated or will start lagging behind other interfaces.

Despite the fact that Spark is based on Scala, I could see somebody slowing the development of the public Scala API before PySpark if some twist of fate required somebody to make that decision.