Hacker News new | ask | show | jobs
by big-chungus4 17 days ago
Adam doesn't use the second derivatives matrix, it uses second moments of the gradient, which is the diagonal of the uncentered covariance matrix, but neither of them are directly related to SVD or singular values anyway.

There is a slight connection where Adam approximates full-matrix Adagrad which computes inverse square root of the convariance matrix, which you usually do using eigendecomposition, but on the covariance matrix SVD and eigendecomposition are equivalent (can easily be converted to each other), so you could use SVD to compute the inverse square root.
2 comments
jmalicki 17 days ago
The second moments of the gradient and the Hessian are absolutely related!

See the Fisher Information, and the Cramer-Rao Lower Bound (an inequality on how much the inverse covariance matrix and the Hessian can differ).

https://en.wikipedia.org/wiki/Fisher_information

link
jmalicki 17 days ago
I didn't find a direct proof earlier, just assertions - I've only seen it proved in textbooks that aren't linkable.

Theorem 1, section 1.3, page 2 shows that the expected variance of the gradient of the loss function and the expected second derivative of the loss function are equal at the minimum. I hate that the ADAM paper did not talk about this, this is something that is hammered into anyone who has taken a mathematical statistics course. This has been an established fact in statistics for well over 100 years.

https://courses.grainger.illinois.edu/ece563/fa2025/Notes11-...

Away from the minimum they can diverge, but there is a close enough connection to make it an extremely useful approximation.

link