|
|
|
|
|
|
by jmalicki
21 days ago
|
|
|
I didn't find a direct proof earlier, just assertions - I've only seen it proved in textbooks that aren't linkable. Theorem 1, section 1.3, page 2 shows that the expected variance of the gradient of the loss function and the expected second derivative of the loss function are equal at the minimum. I hate that the ADAM paper did not talk about this, this is something that is hammered into anyone who has taken a mathematical statistics course. This has been an established fact in statistics for well over 100 years. https://courses.grainger.illinois.edu/ece563/fa2025/Notes11-... Away from the minimum they can diverge, but there is a close enough connection to make it an extremely useful approximation. |
|
|