[lostintangent]

Very dumb question: when you say backed by Akka, what exactly do you mean? That you’ve configured a “binding” between Flint/Spark and Akka, which spins up actor instances (Java/Scala classes?) whenever some events occur? Or that Flint/Spark use Akka behind the scenes?

[closeparen]

They both use Akka behind the scenes. I don’t know the internal details exactly, but they’re both capable of showing you the processing pipeline you’ve asked for as a DAG visualization. I think what’s happening is something like each node on that DAG is an actor, receiving data from the previous node and sending it along to the next. You don’t write that though, you just write map/filter/etc.

[trebecks]

flink uses akka for control plane kind of stuff. data plane stuff, like piping data between tasks, doesn't. check out https://flink.apache.org/2019/06/05/flink-network-stack.html for some (maybe outdated) details.