[arolihas]

Idk to me this is just redescribing what deep neural networks do without actually explaining why anything happens. I guess it "unifies" things but I am kinda over most unifying theories. Everything is Bayesian, everything is a graph or a group or some other fancy geometric structure, everything is a category. Ultimately the best framework is whatever is useful enough to explain what's happening in such a way that a practitioner can manipulate the model towards a desired outcome. In other words, where is the knob? The tool they share may be interesting and I hope to play with it to see what happens at different levels of noise applied to the labels.

[po1nt]

We're still in the era of room-sized-computers-only-scientists-understand era of the neural networks. Knobs and buttons for nerds are slowly coming.

[arolihas]

I agree, which is why it’s too early to make such grandiose claims about deep learning theory.

[ipnon]

A real theory would predict phenomena thus far unseen. We already know about this 4 part taxonomy.

[T-A]

Did you also know about this?

Lastly, we derive an exact population-risk objective from a single training run with no validation data, for any architecture, loss, or optimizer, and prove that it measures precisely the noise in the signal channel. This objective reduces in practice to an SNR preconditioner on top of Adam, adding one state vector at no extra cost; it accelerates grokking by 5x, suppresses memorization in PINNs and implicit neural representations, and improves DPO fine-tuning under noisy preferences while staying 3x closer to the reference policy. [1]

[1] https://arxiv.org/abs/2605.01172