|
|
|
|
|
|
by caravel
2849 days ago
|
|
|
[full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Airflow speaks for itself https://github.com/apache/incubator-superset#who-uses-apache... But hey, it's like anything, you have to do a bit of work to get distributed systems to run at scale. There are now hosted solutions to help with that (Google Cloud Composer and Astronomer.io) |
|
|
:)