[asntbqjkwbaoeuc]

Depending on how you look at it, this idea has been around for a long time. At least since the 1980's it's been known that functional programming can be considered the logical basis of all of math [0]. A hardliner might even say that the notion that all math is functional programming goes back to the 1920s with Brouwer.

In particular, optimization, being a branch of math, is itself a type of functional programming. Similarly, neural networks are functional programs. Programming with differentiable functions as a field on its own goes back at least as far as the 1960s. Closely related ideas like algebraic topology go back to the 1890s (this would be programming with continuous functions rather than differentiable).

The language of type theory is typically the province of computer science. The corresponding language in math is category theory. There are a few mathematicians who have been working on a categorical theory of neural networks. But this is pretty far from a mainstream research area.

Perhaps with people like Colah and Dalrymple pointing out these connections from a more applied point of view, these ideas will pick up steam.

[0] There is are technical caveats here that aren't interesting when talking about math that can be done on computers.