Hacker News new | ask | show | jobs
by krcz 2398 days ago
Spark uses off heap memory via JNI in order to avoid JVM memory management drawbacks. It hides it behind a heavy abstraction, which makes it plausible to use such solution for data processing pipelines, for the smaller applications not so much.

But for most things that is not that much a problem - I'm not saying that JVM memory model makes Scala unusable, just that Spark is not a great example here.