[charleshmartin]

Here's an alternative approach, that actually provides real world results

https://calculatedcontent.com/2019/12/03/towards-a-new-theor...

Using techniques from statistical mechanics and strongly correlated systems, we can compute the average-case-behavior of a real world DNN

We believe we can reproduce some of the results of the NTK by using a Gaussian Random Matrix. But if we use a more realistic, heavy tailed matrix, we get more practical results

A early fork of the theory has been published in JMLR https://arxiv.org/abs/1810.01075

and the empirical results in Nature Communications https://www.nature.com/articles/s41467-021-24025-8

and we have an open source tool , weightwatcher, which can be used in production

pip install weightwatcher

https://github.com/CalculatedContent/WeightWatcher

Please give it a try and let me know if it is useful to you

[thisnormaluser]

Nature Communications is not Nature. But whatever.

[charleshmartin]

That's fine. Here's a few of my online talks:

UC Berkeley / ICSI: https://www.youtube.com/watch?v=6Zgul4oygMc

Stanford ICME: https://www.youtube.com/watch?v=PQUItQi-B-I

and a couple of Mikes's

Institute for Pure & Applied Mathematics (IPAM) : https://www.youtube.com/watch?v=fmVuNRKsQa8

Physics Informed Machine Learning: https://www.youtube.com/watch?v=eXhwLtjtUsI

and our KDD workshop KDD: https://dl.acm.org/doi/abs/10.1145/3292500.3332294

many more talks and papers available

And my favorite, a podcast that has featured LeCun himself:

https://blog.rebellionresearch.com/blog/theoretical-physicis...

[charleshmartin]

Oh and here's a 2 hour deep dive into the theory https://bluejeans.com/playback/s/cyQnC9EZZSB9HlHDjB1Zpu3fs0J...

[chaxor]

I actually have a preference towards Nature Communications over Nature, as Nature Communications actually allows more pages for explanation and as such typically has more rigorous research.

On the flip side, Nature often has bolder claims with less convincing evidence, which often reads similar to pop-sci articles.