[contravariant]

If you can turn it into a bunch of python (or bash) scripts that should be scheduled after each other you could probably use airflow. I'm not saying you should, but it's perfectly doable. Airflow works best when you're free to restart jobs though, that's probably something to keep in mind.

And Airflow is designed to spread tasks between a set of available workers, so while you can make jobs that trigger something remotely through SSH (for example) I'm still looking for a way to have a 'remote worker' that runs airflow jobs on the remote system itself rather than through an SSH connection (mostly because of a rather peculiar use-case where we'd prefer to run jobs on a remote network without two-way communication).

[x87678r]

Thanks, I tried airflow before and gave up because it was more complex to set up than I hoped but will try again with 2.0

[contravariant]

Yeah it has a few components that need to be set up (at minimum a database and the airflow scheduler + worker) to get it to run well.

If you're comfortable with docker-compose you can probably find an example setup and get it running in a few minutes though.