[jmngomes]

More than 90% of the enterprises I've worked with use one of the enterprise tools listed in Gartner's magic quadrant, typically from the Leader quadrant, sometimes from the Niche quadrant, here are a few examples: https://duckduckgo.com/?q=gartner+magic+quadrant+data+integr...

What's actually mind boggling for me, and I wonder if it's a bit of over-engineering, is people going for complex setups (oh, it's just Airflow scripts with k8s and a little bit of SystemD services and configs plus some shell scripts) when there are COTS tools that do more for less engineering cost. Yes, these carry a price tag, but it's usually quite less than paying for engineers to babysit a tool with a ton of moving parts...

[massaman_yams]

Or, you can look at the (say) 6 month sales cycle for an enterprise platform, and the internal political wrangling that may be required to get approval for the line item, as significant hurdles to moving forward with an ETL project. There are some legitimate, and some less valid, reasons for many engineers' (and engineering-driven orgs') bias against commercial options; and yes, sometimes this results in over-engineering.

Airflow can do some things many commercial tools cannot, though, so for some it is the right option.