[TankeJosh]

You can think of BuildFlow as a lightweight alternative to Flink / Spark Streaming. These streaming frameworks are great when you want to react to events in realtime (i.e. you want to trigger some processing logic every time a file is uploaded to cloud storage). Dagster is more focused on scheduling jobs, and might be a good fit if you have some batch jobs you want to trigger occasionally.

BuildFlow can run a simple PubSub -> light processing -> BigQuery pipeline at about 5-7k messages / second on a 4core VM (tested on GCP’s n1-standard-4 machines). For your case, you might be able to get away with running on a single machine with 4-8 cores.

I’d be happy to connect outside of HN if you’d like me to dig into your use case more! You can reach me at [e-mail]

edit: You can also reach out on our discord: https://discordapp.com/invite/wz7fjHyrCA

[vosper]

Thanks for that. Sounds like it might fit what we want. I'll reach out if I have any more questions.

Are you tied to GCP services like pubsub and BiqQuery? We're in AWS, not GCP.

[TankeJosh]

AWS support is in the queue, but we only have GCP services at the moment. What services on AWS do you need access to? We can move them to the front of the queue to help out.

Feel free to reach out even if this doesn’t work with your timeline. I might be able to help you come up with another solution, and I’m always interested to hear new use cases!