[logicfiction]

I don't think Scala Native would be a good fit with Apache Spark. Starting with their work in 2.0, the Spark framework already started doing things like working with sun.misc.Unsafe for direct memory control over certain operations. In general they put a lot of effort into getting as close to metal and hyper-optimizing the framework itself. Given that, trying to do the same yourself on top of that framework seems counter-intuitive in my opinion.

[lobster_johnson]

Surely a goal with Scala Native is performance and things like "direct memory control" (why else move away from the JVM?), which would very much dovetail with the work being put into Spark?

[lmm]

The JVM has had native-like throughput for long-running batch processing jobs for years - of course you can always tune further, but it already fits nicely with what Spark is used for. Scala Native makes more sense for things like command-line utilities that need to start up quickly, or in the future maybe real-time systems that need to avoid GC.