|
|
|
|
|
by tedmiston
2958 days ago
|
|
I'm not totally sure how to answer the question you're asking. The local dev cycle is much faster than deploying and running remotely. Locally the cluster runs two main projects, both systems that handle user-defined tasks. One is a Kafka-based clickstream event processor, the other is an Airflow-based workflow ETL system built on Celery. Exact resource requirements depend on the requirements of the task of the moment and how fast you want it to run. Of course there is the related tooling and services as well such as ZooKeeper, Celery, Redis, Postgres, Grafana, Prometheus, etc. I haven't tried this on a tmux type setup on a remote box for my "local" dev before. The cost to run it on a cloud deploy is non-trivial. |
|