[voidhorse]

But there are direct instances that suggest ChatGPT is limited in exactly the way Chomsky and his colleagues suggest. It has made plenty of mistakes that someone who understands the rules of logical reasoning would not make.

Furthermore, all the hallucinatory effects (making up APIs, making up references) would suggest it really is still just statistical output...

[mpweiher]

It hallucinates all the time.

We humans just don't have a good intuition for what hallucination with an insane data set looks like.

[voidhorse]

Totally, I also think there's a huge vicious cycle risk with these tools.

The more we rely on them, the more our own mental model of existence and knowledge will adapt to become little more than statistical aggregates of data. Science has already trended this way in certain areas. We used to reason from deductive principles, look for data, and fit data to both inductive hypotheses and deductive principles. Steadily we've come to care less and less about the deductive part and think of everything in terms of probabilities and statistical models. While this is not bad per se, I do think our growing ignorance of deductive reasoning causes us to miss some important aspects of existence.

[calf]

That reminds me of the whole most real numbers are irrational, in fact the vast-vast majority of points on the number line you see before you, are not rational numbers. Humans don't have a good intuition for maths. So guess what happens when you brute-force a neural network machine into an AI?

[bhouston]

It has limitations for sure. I asked it some questions about an API and it told me it had a specific capability, which I didn't think it had. Then I asked it how it knew that and it gave me an URL to docs and a quote. While the URL existed, the quote from that page didn't, and it never did exist (I checked.) It was incredibly strange to see it confidently tell me things that were clearly statistical extrapolations.

So yeah, ChatGPT is awesome, but it doesn't differentiate reality from its statistical extrapolations.

I think there has to be a way to possibly add a module on top that somehow is trained to identify reality-based content and when it is making up likely scenarios. Humans are capable of both of these modes but we differentiate between them. ChatGPT is capable of both of these modes, just doesn't have the differentiator yet.

[jeisc]

AI has no way to distinguish reality from make believe because it lives on a silicon chip in a made up world fed information by us imperfect creatures.