|
|
|
|
|
by contravariant
1590 days ago
|
|
I get what you're saying but trying to make sure your code will run on airflow is the wrong way of thinking about it IMHO. You should be trying to get airflow to make sure your code runs (could be in airflow, could be anywhere else). A lot of the stuff we do with airflow is just basically sending commands and looking at the result (and handling any errors), this part is generic enough that you usually only need to implement it once for whatever platform your code is running on. The tricky bit is when your DAG crosses platforms, but that's always a problem. If anything it's easier to solve when the tool scheduling tasks isn't part of the platform (note however that airflow is not a tool for solving dataflow, though some glue code in python does often work wonders). |
|