[ewhauser421]

Why not Apache Beam? It doesn’t require Spark or Flink

[miraculixx]

Thanks for the question. Beam has a more complex execution model and AFAIK also needs some executor environment like Spark to really parallelize workloads. Given a mongodb that all producers and consumers can attach to, minibatch runs anywhere.

[ewhauser421]

(Beam has a DirectRunner which is just in memory)