[niwtsol]

Yeah, I love this — pure Python with cron or periodic tasks (e.g., Django) works great. Celery task for parallelization, and if you pipe logs/alerts into a Slack channel, you can actually get really far without needing a "proper" orchestration layer.

I recently took over an Airflow system from a former colleague, and in our case, it’s just overly complex for what’s really a pretty simple data flow.

[mettamage]

I don’t know much about airflow

But isn’t it just also python with cron?

[datadrivenangel]

Airflow is python with cron, and the option for very sophisticated and useful orchestration tools, like retries, dependencies, etc. All the stuff you'll end up rolling yourself as your simple scheduled tasks grow.