| HP OO is an entire system that has an XML structure command structure to customize the job that you're working on. It has a GUI that is used to build out the flows, run and test. It has a backend system to audit, admin, and visualize the current process, and it has workers to scale out the work. It's a bit of a more mature setup. The claim that the setup of your workflow has to be code isn't necessarily a good thing. Your recipe should be descriptive, not imperative. ---- To answer your response: 1. Stability: I was working with apache-airflow 1.9 (last release: Jan2018) 1.10 was just released 2 days ago. I frequently had issues where deleting more than 3 tasks would cause that mushroom cloud error message. Also, I've had cases where the task could max out on memory and take the whole system down. 2. 1.9.0: Stopping jobs: I saw this issue where that a task would be running, I would stop+delete the task and start a new one. I frequently saw the case where I had to wait a while for the triggered dag to continue running. 3. Python3.7: Yes, it was addressed on the PR. However, for things like that we (the users) need a quick turn arround/hotfix for stuff like that. It got released late (lets say 27 June, and the latest version 1.10 was 28 August [with a 7month gap]) It's just painful to have this upgrade just break something internally in Airflow. 4. From what I've seen in situations where the work for the task is huge is that there is an expectation of the task to handle the workload and splitting up the workload it's self. (Since you can't define a span out of the tasks based on the workload) That's no beuno. Timeouts:
From what I have seen there are issues where the next dag run scheduled can interfere with the last one. This is an issue given the timeouts, retries, and reoccurring schedules. (yes you can say.. that's user's choice.. however, workloads and performance can change without notice) ---- Another issue I had: There is no way to trigger a task and it's depending tasks without triggering the whole dag. This makes long-running dags with lots of tasks difficult to debug and test. Also there is a slight difference between airflow run (task) and test. Sometimes you use one vs the other. |