Hacker News new | ask | show | jobs
by dimberman 2006 days ago
I would say at this point Airflow is leaning pretty heavily on being a data tool. I wouldn't recommend it for something like CI/CD for example. Do you have a use-case in mind?
1 comments

Every corporate job I've worked at had autosys or control-m to schedule linked jobs - not just data stuff. Would be good to replace those systems.
If you can turn it into a bunch of python (or bash) scripts that should be scheduled after each other you could probably use airflow. I'm not saying you should, but it's perfectly doable. Airflow works best when you're free to restart jobs though, that's probably something to keep in mind.

And Airflow is designed to spread tasks between a set of available workers, so while you can make jobs that trigger something remotely through SSH (for example) I'm still looking for a way to have a 'remote worker' that runs airflow jobs on the remote system itself rather than through an SSH connection (mostly because of a rather peculiar use-case where we'd prefer to run jobs on a remote network without two-way communication).

Thanks, I tried airflow before and gave up because it was more complex to set up than I hoped but will try again with 2.0
Yeah it has a few components that need to be set up (at minimum a database and the airflow scheduler + worker) to get it to run well.

If you're comfortable with docker-compose you can probably find an example setup and get it running in a few minutes though.