Hacker News new | ask | show | jobs
by jmmcd 638 days ago
Yes, he's handwaving in this general area, but no, he's not really relying on the UAT. If you talked to most NN people 2 decades ago and asked about this, they might well answer in terms of the UAT. But nowadays, most people, including here Altman, would answer in terms of practical experience of success in learning a surprisingly diverse array of distributions using a single architecture.
1 comments
ansk 637 days ago
I think that while researchers would agree that the empirical success of deep learning has been remarkable, they would still agree that the language used here -- "an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data)" -- is an overly strong characterization, to the point that it is no longer accurate. A hash function is a good example of a generating process which NN + SGD will not learn with any degree of generalization. If you trained GPT4 on an infinite dataset of strings and their corresponding hashes, it would simply saturate its 100 billion+ parameters with something akin to a compressed lookup table of input/output pairs, despite the true generating process being a program that could be expressed in less than a kilobyte. On unseen data, it would be no better than a uniform prior over hashes. Anyways, my point is that people knowledgable in the field would have far more tempered takes on the practical limits of deep learning, and would reserve the absolute framing used here for claims that have been proven formally.
link
jmmcd 633 days ago
Certainly, it's an exaggeration / simplification. I don't feel it is really a dishonest one, in context [*]. It feels weird for me to "defend" him here, because in general Altman is a stupendous, egregious, world-leading liar.

[*] Ok, we can differ on that. My feeling is partly because the types of distributions that can't be learned - eg hash functions - are generally the kind of functions we don't really want to learn. Underneath this are deeper questions related to no free lunch and how "nice"/"well-behaved" this universe is.

link