[bunderbunder]

On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.

[vvladymyrov]

For python guys - pyspark is now also installable from pip as a package (it include some .jars so it is ~100Mb size python package). So my team for local unit tests installs pyspark as a package.

[MichaelRys]

These are all good points. Debugability and general support for the development lifecycle are important. We are definitively working on providing first class development experiences for .NET developers. .NET for Apache Spark is already available as a nuget package for local install. We are currently working on adding support to VS Code, Visualstudio etc. Feel free to provide us your preferred dev platform. [Disclaimer: I am Program Manager for the .NET for Apache Spark effort]

[bunderbunder]

Thanks for the response!

FWIW, I was speaking specifically to being able to run Spark, and manage its lifecycle, all inside the same process as the unit test code. Which is something that I'll openly concede isn't much more than a fun party trick for most people's purposes, but it does happen to serve me well.

In a past life, I was involved in data engineering at a .NET shop, and being able to migrate parts of our process to something like Spark without having to rewrite or otherwise severely damage it would have made me very happy. Even better if I could stay inside Visual Studio, and bang on it from an F# interactive session.

Wild speculation, but if you can produce type providers that know how to tame `DataSet[Row]`, you might have some nonzero number of F# hipsters like me kissing your feet.

(Or not. Like I said, my perspective on Spark is unusual.)

[MichaelRys]

Thanks... More idiomatic F# support is on the roadmap