[jmalicki]

ADAM is related if your second derivative matrix happens to be diagonal.

Of course, it takes about 5 minutes to show that any DNN is going to have very very high magnitude off-diagonal terms by the way it's constructed, so pretending that a diagonal approximation is close enough is crazy.

[big-chungus4]

Adam doesn't use the second derivatives matrix, it uses second moments of the gradient, which is the diagonal of the uncentered covariance matrix, but neither of them are directly related to SVD or singular values anyway.

There is a slight connection where Adam approximates full-matrix Adagrad which computes inverse square root of the convariance matrix, which you usually do using eigendecomposition, but on the covariance matrix SVD and eigendecomposition are equivalent (can easily be converted to each other), so you could use SVD to compute the inverse square root.

[jmalicki]

The second moments of the gradient and the Hessian are absolutely related!

See the Fisher Information, and the Cramer-Rao Lower Bound (an inequality on how much the inverse covariance matrix and the Hessian can differ).

https://en.wikipedia.org/wiki/Fisher_information

[jmalicki]

I didn't find a direct proof earlier, just assertions - I've only seen it proved in textbooks that aren't linkable.

Theorem 1, section 1.3, page 2 shows that the expected variance of the gradient of the loss function and the expected second derivative of the loss function are equal at the minimum. I hate that the ADAM paper did not talk about this, this is something that is hammered into anyone who has taken a mathematical statistics course. This has been an established fact in statistics for well over 100 years.

https://courses.grainger.illinois.edu/ece563/fa2025/Notes11-...

Away from the minimum they can diverge, but there is a close enough connection to make it an extremely useful approximation.