[thatguyknows]

As mentioned, much of our discussion is rote parroting. I can usually go into any hackernews thread and roughly know what the top discussions are going to be. It's not surprising that an AI trained on a large portion of the internet would thus look human like.

If you really poke at GPT, you begin to realize it's fairly shallow. Human intelligence is like a deep well or pond, where as GPT is a vast but shallow ocean.

Making that ocean deeper is not a trivial problem that we can just throw more compute or data at. We've pretty much tapped out that depth with GPT4 and are going to need better designs.

This could only take half a decade or it could be half a century. Plenty of enterprises stagnate for decades.

[p-e-w]

> Making that ocean deeper is not a trivial problem that we can just throw more compute or data at.

You can't possibly know that, given that we don't actually understand how LLMs work on a high level.

> We've pretty much tapped out that depth with GPT4

GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?

[thatguyknows]

Sam Altman said it himself. He seems like a reasonable source.

If you're familiar with other fields of AI, adding more and more layers to ResNet was the hotness for awhile, but the trick stopped working after awhile.

[steveBK123]

Exactly, and OpenAI has been around nearly 8 years, consumed huge amount of data with tons of compute. They are just showing us the product now.

It is possible they've reached some 80/20 point and he is pretty honest about how much more extendable the current approach really is.

Would explain going to congress and asking for regulation (of their not-quite-there-yet competitors who they want a regulatory moat against).

[famouswaffles]

https://news.ycombinator.com/item?id=36017977

[famouswaffles]

Altman didn't really say that. Reading what he actually said rather than a headline, He was alluding to economical walls. He didn't say anything about diminishing returns on scaling. And if anything, the chief scientist, Ilya thinks there's a lot left to squeeze.

[gmerc]

Sure Sam Altman, the lying CEO of a tech company (they all do) should be listened to on this matter but not on the part where he thinks AGI within reach using his approach. Selective hearing.

[LouisSayers]

> You can't possibly know that, given that we don't actually understand how LLMs work on a high level.

It's a fair assumption to make however - basically 80/20 rule.

AI research isn't a new thing and I bet you could go back 40/50 years where they thought they were about to have a massive breakthrough to human level intelligence.

> GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?

I'm guessing from actually using it.

GPT4 is super impressive and helpful in a practical way, but having used it myself for a while now I get this feeling also. It feels a bit like "it's been fed everything we have, with all the techniques we have, now what?"

[cornel_io]

There are dozens and maybe hundreds of different approaches that could theoretically get around the limitations of GPT4 that merely haven't been trained at scale yet. There is absolutely no lack of ideas in this space, including potentially revolutionary ones, but they take time and money to prove out.

[LouisSayers]

I'm sure there are lots of ideas, but it doesn't mean they're any good or will necessarily transform AI to the next level.

It's going to take time to figure out what works and what doesn't.

There's a reason why Sam Altman is saying they're not training GPT5, and it's not because they think GPT4 is good enough.

[galaxyLogic]

> ... we don't actually understand how LLMs work on a high level.

Are you saying that people who created ChatGPT don't understand how it works? Or that we the rest of people don't?

[famouswaffles]

Training a model doesn't mean you understand what the neurons actually do to influence output. Nobody knows that. That's where the black box analogies come in. We know what goes in the box and what comes out. We don't know what the box is doing to the data

[OrderlyTiamat]

> Making that ocean deeper is not a trivial problem that we can just throw more compute or data at.

I'd say this is immediately counterindicated by the available evidence. Gpt2 was hopeless for anything other than some fun languagw games like a bot replica of a subreddit or trump. 3.5 is much much bigger, and has semi competent but limited reasoning abilities.

Gpt 4 is a vast improvement over 3.5 in various reasoning tasks. Yes, a priori I would have agreed with you that this has to stop somewhere, but not anymore. I would need to see some data of post gpt4 models to believe you.

[atleastoptimal]

The apparent shallowness IMO is due to the lack of long term memory and limited context. In terms of depth, the depth of the human mind is fairly limited. Put any average human to task on any creative endeavor and it's surely a regurgitation of things they've seen. The high standard of true creativity that people hold LLM's to is only a capability of a small minority of humans.

I'd wager it's far more likely 5 years than 50 LLMs get to the full depths all humans are capable of. Simply compare the state of LLMs today vs 2018.