[Fiahil]

> data analysis/engineering as a field seems new and immature enough that, in my humble opinion, we should be focusing on developing good practices and theory, instead of deprecating existing (and pretty recent) tech at an ever increasing pace.

I disagree with you, data engineering as a field has been there for a very long time. Good practices exists and are good enough to accommodate for new ones, like MLOps and data versioning.

However for every great DE setup, you can find at least ten other that are complete pile of shit, featuring mission-critical scripted SQL reports that no one understand anymore and closed source orchestration products with millions-dollars support contrats that only one person has access to.

As always, tooling is rarely an issue. Data Engineers are rarely working on the overall "big picture" and are often given tasks without context. Embedding data engineering with product and infrastructure teams are the solution to that issue.