[ddragon]

It's ok, it seems at this point the focus is in creating the tools to better allow the exploration, including making the entirety of a programming language valid syntax for building any model (supporting the AD). There are the efforts in Julia Zygote and the Tensorflow for Swift that I know of.

I think the differentiable forth example in the article is interesting in the context, since it has a differentiable program with gaps, and it uses the universal approximation property of a neural network to fill them. When your code is differentiable, it's possible to embed ML models, perhaps to learn a part of an equation when you already know most of it, and which otherwise would have too large of a search space. You might even have, like other commenter said, a compiler being smart enough to rewrite the AST to reduce convergence problems (which seems to be the main problem with such models). Or you could download libraries with pretrained models/architectures in the same way you have any regular program library to embed in deeper systems.

Though I honestly can't tell if any of those are actually valid pursuits or I'm misunderstanding the possibilities.