[necubi]

This isn't quite correct (I'm the creator of Arroyo). We use DataFusion to implement parts of our SQL support (in particular the planner and the expression interpreter) but we have our own dataflow and operators. By contrast Synnada[0] is directly built on DF.

A contrast between Arroyo and systems like Benthos and from what I can tell ArkFlow, is that Arroyo is a "stateful" stream processing engine, which means that we can support things like windows, aggregates, and joins, with exactly-once semantics and fault tolerance, at the cost of significant additional complexity[1].

[0] https://www.synnada.ai/ [1] https://www.arroyo.dev/blog/stateful-stream-processing

[chenquan]

Arroyo has been designed with more comprehensive consideration.

[chenquan]

Sorry, please forgive me for not knowing Arroyo completely.

[necubi]

No worries! We definitely rely heavily on DF (it’s an incredible project!). Part of what makes it so great is its modularity—it’s a toolkit for building sql systems, which is extremely cool.

[chenquan]

Yes, whether it is DataFusion, Arroyo, or Bentos, these open source products have made me profit a lot.

[esafak]

That made me chuckle. May you profit in all senses.