I guess when GP says "scientific machine learning" he means "research into machine learning algorithms", which is typically done in academic circles, no?
Then when someone's discovered something new and useful, the focus turns towards making fast and readily available implementations in commonly used frameworks. Like Torch, which you can use both from PyTorch and from Julia.
No, they mean traditional scientific computing (differential equations and the like) merged with machine learning techniques. It's extremely useful in e.g climate modeling. It let's you encode domain knowledge with additional fudge terms approximated by neural networks.
Yeah, this is what I meant. It’s virtually impossible for Python-based libraries to catch up to the huge number of differential equation solvers that are available in Julia, and that work on a GPU, and that can be automatically differentiated through, and that all work through a common interface, and that all work around a well defined ecosystem.
More broadly, I’d say Julia has more potential than Python when it comes to automatic differentiation. JAX is really cool, but Julia’s automatic differentiation tools are targeting the whole language without modification and restriction (outside those imposed by calculus). Potentially some really fancy stuff could get embedded in neural nets beyond differential equations and with relatively little effort.
I find that claim highly dubious, solving differential equations rarely yields to completely generic methods and there are plenty of solver suites in other languages, among them C++ and wrappers for Python. To use scientific machine learning you don't necessarily implement all methods under the sun, just a method appropriate for the domain at hand.
Implementing automatic differentiation through C++ solvers or Python wrappers is going to be exceptionally difficult. If you want a generic toolbox like torch for scientific machine learning you would want every method under the sun.
That’s the moat Julia has here. A full rewrite of the differential equation ecosystem wasn’t necessary in order to support automatic differentiation.
I do not know of another differential equations framework nearly as polished, performant, and feautere rich as the one in Julia, in any language. I actually know of no other framework that has both modern solvers and autodiff working together. The only other autodiff solvers I know of are ancient RK4-like solvers implemented in things like Tensorflow. It is hard to overstate how monumentally enabling such tools are for old-school science and they currently exist only in Julia.
Sundials (https://computing.llnl.gov/projects/sundials) is arguably more feature complete, has many optimised implementations scaling to super computers and supports both adjoint and forward sensitivity analysis. Btw. several "research" papers at machine learning conferences on "Neural Differential Equations" presented "discoveries" covered in the documentation of Sundials https://sundials.readthedocs.io/en/latest/cvodes/Mathematics....
while there are c++ solvers, they are missing the flexibility needed to solve many types of problems efficiently. they almost all use finite differencing, which is inaccurate and slow, and they rarely let users customize the linear solve and nonlinear solve algorithms. What the Julia diffeq suite is working towards is complete control over solver options, while picking good defaults when not customized. They let you pick forward and backward mode AD (or a combination of the two), problem specific preconditioning methods, custom root finding algorithms (including gpu based linear solvers), arbitrary precision methods for verification (or if you're just paranoid), interval arithmetic, linear uncertainty propegation, and more.
As far as I'm aware there is considerable prior art, since solving this problem has been relevant since at least the 70s. Granted the barrier to entry was far higher then, then now, but we are talking about scientific / engineering applications and there the prior investment in specialised tools has always been justified. A contemporary example is http://www.dolfin-adjoint.org/en/latest/, but there are others. Sundials for example also both supports completely custom linear and non-linear solvers in conjunction with adjoint sensitivity analysis.
Then when someone's discovered something new and useful, the focus turns towards making fast and readily available implementations in commonly used frameworks. Like Torch, which you can use both from PyTorch and from Julia.