[jebarker]

I’m not convinced that useful mathematical foundations will be found anytime soon. Neural nets exist because we want to make decisions in a world that is so noisy, complex and chaotic that we can’t satisfy the requirements of more rigorous analytical frameworks. It seems to me that the irreducible complexity is in the real world, not the neural networks.

[ontouchstart]

Never underestimate the power of mathematics. On the other hand, a lot of mathematical breakthroughs in the history are not found by people doing mathematics but in physics and engineering. Calculus, statistics, information theory, etc.

Even for computer science, take a look at Turing Award from 1966 [0], we will see how short sighted we are if we only follow the trend. Time will tell and smart people will find new path.

[0]: https://amturing.acm.org/byyear.cfm

[jebarker]

I have a background in mathematics, I believe in mathematics, but I don't believe in blind faith. Physics gave us statistical mechanics precisely because it's impossible to measure, model and predict the behavior of every individual particle in real-world systems. My gut feeling is that a mathematical theory of LLMs is more likely to look like statistical mechanics than something that tames chaos. That certainly doesn't mean that theory wont't open new doors though that we haven't currently thought of.

[ontouchstart]

My point is that new mathematics will eventually come out of this mess. [0]

[0] https://youtu.be/2Op3QLzMgSY?t=75