[magicalhippo]

The previous post in the series[1], helpfully linked to in the introduction, has a lot more details of the underlying concepts.

It also set my mind wandering to the not-technically-related functional derivatives[2], where you vary the function slightly rather than the argument value.

I'm not great at math, but I do love this what-if exploration you can do in math. Due to the various proofs underlying it all it seems sometimes more fruitful than similar exploration in programming, where one might quickly stumble upon obscure compiler errors or similar obstacles.

[1]: https://nosferalatu.com/DerivativesLogarithmsTransforms.html

[2]: https://en.wikipedia.org/wiki/Functional_derivative

[abhgh]

If you're looking for interesting derivative-adjacent ideas, I would also recommend Clarke derivatives [1]. They occasionally show up in ML papers, e.g., [2], [3]. Unrelated bu tangential, another place where you need derivatives but don't have access to them (standard or otherwise) is in the area of black-box optimization. Within this area, Bayesian Optimization (BayesOpt) has picked up quite a bit, which I've successfully used quite a bit in my work - I've an introduction here [4]. There is also a good book available online for free on the topic [5].

[1] https://en.wikipedia.org/wiki/Clarke_generalized_derivative

[2] https://proceedings.neurips.cc/paper/2021/file/70afbf2259b44...

[3] http://proceedings.mlr.press/v202/lee23p/lee23p.pdf

[4] https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs

[5] https://bayesoptbook.com/

[ogogmad]

Functional derivative : Frechet derivative :: gradient : directional derivative