[pdinny]

The post feels like a bait-and-switch in the sense that it presents itself as about Airflow's shortcomings but focuses mostly on problems that Airflow doesn't attempt to solve.

Airflow can certainly be frustrating and it doesn't solve _all_ workflow orchestration problems. Surely the same thing can be said of many tools? This seems mostly like a mismatch of expectations.

[evrydayhustling]

FWIW the author is pretty direct about this. After the cute beginning, he basically says that his problem is with Airflow's scope, not it's execution.

[pdinny]

Hence the bait and switch. IMHO increasing the scope of Airflow (or any tool) is a challenging proposition. Would you rather use very few mega-tools with very broad scope (and potentially more challenging domain to navigate) or fewer more specialised tools that interoperate well together?

Obviously there are trade-offs with either approach, but then I'd argue that making Airflow solve more problems will introduce more trade-offs too.

[peteradio]

Having a poor scope is a problem because people will just choose not to use you.

[cturner]

When we engage in complex work, it is important to keep our options open so we can direct our best effort to the hardest problems.

It is rarely clear what the hard problems will be when new to a domain. Only as scale kicks in.

We are constantly pitched frameworks that sell themselves as a good approach to a domain, but then obstruct engagement with the hardest problems when it matters. The developer becomes captive of the system that claimed it would steer them right.

This is particularly true of fields where the hard problems are integration problems which, by their nature, cannot be outsourced to frameworks.

[MontyCarloHall]

100% agreed. By analogy, an article titled “pthreads’ problem” should be about shortcomings in the POSIX multithreading model, not an article saying that the implementation of machine-level parallelism is irrelevant because Kubernetes exists.