[rotten]

I've recently had the sales team from Magniv.io pestering me to try it as an alternative to Airflow with a "shift left" perspective for automating jobs. I wasn't convinced enougy by the value prop to dive deeper. I think it was a language problem - I was just having trouble understanding and relating to the problems they solve and then figuring out whether or not I have those problems too.

I'm running into the same issue with this guy's post, although a little less so. The question he seems to ask is "With a complex pattern of data flows, if something breaks, how do you recover?" His argument is that Airflow does not offer enough visibility into the full data trace nor enough tools to apply recovery rules for repairing broken bits.

I think I agree, but prometheus doesn't really solve that. Nor necessarily does better management of automated job queue backlog management and job retries.

He also complains about some syntax and design choices that predate MyPy and Pydantic and modern Async Python coding. Those seem fairly easy things to drag Airflow forward with in future releases.