[YeGoblynQueenne]

>> A lot of what Sutskever says is wild. But not nearly as wild as it would have sounded just one or two years ago. As he tells me himself, ChatGPT has already rewritten a lot of people’s expectations about what’s coming, turning “will never happen” into “will happen faster than you think.”

In the '90s NP-complete problems were hard and today they are easy, or at least there is a great many instances of NP-complete problems that can be solved thanks to algorithmic advances, like Conflict-Driven Clause Learning for SAT.

And yet we are nowhere near finding efficient decision algorithms for NP-complete problems, or knowing whether they exist, neither can we easily solve all NP-complete problems.

That is to say, you can make a lot of progress in solving specific, special cases of a class of problems, even a great many of them, without making any progress towards a solution to the general case.

The lesson applies to general intelligence and LLMs: LLMs solve a (very) special case of intelligence, the ability to generate text in context, but make no progress towards the general case, of understanding and generating language at will. I mean, LLMs don't even model anything like "will"; only text.

And perhaps that's not as easy to see for LLMs as it is for SAT, mainly because we don't have a theory of intelligence (let alone artificial general intelligence) as developed as we do for SAT problems. But it should be clear that, if a system trained on the entire web and capable of generating smooth grammatical language, and even in a way that makes sense often, has not yet achieved independent, general intelligence, that's not the way to achieve it.

[nopinsight]

The architectures we know of so far have not been sufficient to achieve AGI with just text and image data. Humans and higher animals learn with much richer modalities than those two and probably would not be nearly as intelligent if forced to learn with just text and images. There are already ongoing efforts to train models with other modalities. Latest foundation models already go beyond pure LLMs.

Your reasoning above doesn’t mean some improvements to the current architecture(s) coupled with richer data would not be sufficient to achieve AGI.

There’s also a possibility OpenAI has recently achieved a yet undisclosed breakthrough.

Sam Altman at the APEC Summit yesterday:

"4 times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we push the veil of ignorance back and the frontier of discovery forward”

https://twitter.com/SpencerKSchiff/status/172564613068224524...