[dandelionv1bes]

I’ve been slowing crunching through Math for Deep Learning, so spent a fair amount of time looking at Hessian matrices + second order optimisation. I’ve been slowly reading this book for a year, so stopping to do most of the math by hand each time. One chapter to go!

Then I was sick all last week, so ended up down a rabbit hole about the current card collecting bubble (right word?). Super interesting.

[dapperdrake]

Where do Hessians come into play for neural networks? It seems like they just use auto-diff to compute the Jacobian or the gradient for backpropagation.

The theoretical results sometime look at the second order derivative.