Dependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.
Do you know what other ecosystem comes closest to the existing in Python? I've heard good things about Julia.
13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends).
There is also a lot of development in Elixir ecosystem around the subject [1].
I don't think the ML community has an appetite for learning a different language.
There are Microsoft-backed F# bindings for Spark and Torch, but no one seems interested. And this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance.
The answer will probably be JavaScript.
Everyone already knows the language - all that's missing is operator overloading and a few key bindings.
> There are Microsoft-backed F# bindings for Spark and Torch
> this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance.
For exactly the reasons your mentioned, i feel like F# would have been the perfect match for both MLE/ETL(spark pipeline) work and some of the deep learning/graph modelisation such as pytorch.
Saddly, even from MSFT, the investment in F# as dried up
Since then the whole python ecosystem has gotten worse.
We are building towers on quicksand.
It's not about python, it's about people who don't care about dependencies.