[jmcgeeney]

The difference between humans and sophisticated stochastic parrots is reason. The most common kinds of mistakes that chatGPT currently makes are when it says things that are not simply wrong, but don't make sense. Perhaps it will be possible to emulate reason with enough data, training, parameters, etc, but without some representation of the ability to understand what you don't know, what you know, and what follows from those things consistently, I wonder if these kinds of models will ever become truly reliable.

[ldhough]

Although even in this example it does say some things that are simply wrong:

"Some programming languages like TypeScript and Swift use a colon for both variable type annotations and function return types."

This is incorrect for Swift, which uses the same arrows as "TenetLang" for function return types. Actually the first thing I thought looking at the example code was "looks swifty but not quite as well designed."

[galaxyLogic]

> some representation of the ability to understand what you don't know, what you know, and what follows from those things consistently

Right, that is truly what the chatbots are lacking. They can fool some of the people some of the time, but they can't fool all of the people all of the time.

Their creators - not the chatbots themselves - are trying to "fool" people into believing that the chatbots would have as you say "ability to understand what you don't know, what you know, and what follows from those things consistently".

[raincole]

It's totally possible that humans don't do reason. It's possible that the parrot in our brain makes the decision, and then the "frontend" of our mind makes fake reasons that sound logical enough.

But it's just a possibility, and I don't find it's particularly convincing.

[vidarh]

Look up split brain experiments [1]. Basically, in patients where the corpus callosum is severed to some degree, the two halves of the brain have limited communication. Since the two halves control different parts of the sensory system, you can provide information selectively to one or the other half of the brain, and ask that part of the brain to make choices. If you then provide the other half of the brain the wrong information, and ask it to reason about why it made a choice, the other brain will happily pull a ChatGPT and "hallucinate" a reason for a choice neither that half nor the other half of the brain ever made.

While that does not prove that we never apply reasoning ahead of time, it is a pretty compelling indication that we can not trust that reasoning we give isn't a post-rationalisation rather than an indication of our actual reasoning, if any.

[1] https://en.wikipedia.org/wiki/Split-brain

[kqr]

No but that does happen a lot of the time. The difference is that we can choose to engage the deductive engine to verify what the parrot says. Sometimes it's easier to do so (what's 17×9?) and sometimes it's harder (a ball and a bat costs $1.1 and the bat costs $1 more than the ball, what does the ball cost?)

[vidarh]

You can ask ChatGPT to show it's working too. It's likely, for many things, (as it is with humans a lot of the time) that the way it does it when it walks you through the process is entirely different to how it does it when it just spits out an answer, but it does work. E.g. I asked it a lot of questions about the effects of artificial gravity using a rotating habitat a few days ago, and had it elaborate a lot of the answers, and it was able to set out the calculations step by step. It wasn't perfect - it made the occasional mistakes - but it also corrected them when I asked follow-up questions about inconsistencies etc. the same way a human would.

(Funnily enough, while both your example questions are easy enough, I feel I was marginally slower on your "easy" question than your "harder" one)

[kqr]

Maybe we're getting close to what makes me doubt the utility of LLMs. Much like humans, they are quick to employ System 1 instead of System 2. Unlike humans, their System 1 is so well trained it has a response for almost everything, so it doesn't have a useful heuristic for engaging System 2.

[vidarh]

For me the utility is here today. Even if I have to carefully probe and check the responses there are still plenty of things where it's worth it to me to use it right now.

[RHSman2]

Fast thinking/slow thinking by Daniel Kannerman (and the Unlocking Project) is a book people need to read and understand. When we slow think, we work things out, when we fast think we are parrots.

[AnimalMuppet]

Hmm. If you want to argue that ChatGPT has half of what an AI needs, I could buy that a lot more than I buy "ChatGPT is the road to AGI".

Do inference engines have the other half? Or Cyc plus an inference engine? Can that be coupled to ChatGPT?

My own (completely uninformed) take is that such a coupled AI would be very formidable (far more than ChatGPT), but that it will be very hard to do so, because the representations are totally different. Like, really totally - there is no common ground at all.

[RHSman2]

I didn’t say half. I just said we have 2 ways of thinking.

[AnimalMuppet]

I just meant "half" in the sense that there are two major chunks that are needed. I did not mean that each "half" was used as much as the other, or was as much work to implement.

[ummonk]

I’m sure a stochastic parrot model could be trained to exhibit reasoning, but the issue is that there isn’t any automated loss function which can discern whether the output of a large language model exhibits reasoning or is illogical. When you train based on text similarity, it will have a hard time learning logic, especially given the amount of illogical writing that is out there.

[AnimalMuppet]

Hmm. That's classically the job of an educator - to choose good training material, not just to let students read the internet all day. Would a ChatGPT trained on a carefully curated reading list do better than one trained on a wider reading list?

But it's more than that. A good educator teaches students to evaluate sources, not to just believe everything they read. As far as I can tell, ChatGPT totally lacks that, and it hurts.

[RHSman2]

And our reason is controlled by emotions, ego, intellect, etc, etc: our sentience.

Llm’s are the language model. That is it.

Now we have gone down this path to a very useful tool what’s next is connecting it to something that can understand context and memory.

It’s not so hard to imagine being connected to a knowledge graph of all the things and this evolving into a very capable AI.

It’s like Google v1 compared to Google now. Key words versus semantic seo

[midoridensha]

>The most common kinds of mistakes that chatGPT currently makes are when it says things that are not simply wrong, but don't make sense.

So, like the average voter?

[vidarh]

You're being a bit glib, but I tend to agree that a lot of people seriously overestimate the reasoning done by the average human.

I think GPT is on the other hand both over- and underestimated because it speaks well but often makes reasoning mistakes we only expect of someone less eloquent, and it throws us.

If it had come across as an inquisitive child, we'd have been a lot more overbearing of it's "hallucinations" for example, because kids do variations over that all the time.

At the same time, it can do some things most children would have no hope of.

It's a category of "intelligence" we're unfamiliar with, additionally hobbled with no dynamic long term memory.

[vidarh]

We have far too insufficient knowledge of how human reasoning work to claim to know we are more than stochastic parrots with vastly more context/memory. It's way too early to claim there's some qualitative difference.

(And humans are not very reliable; more than current models, sure, but still pretty bad)

[HervalFreire]

Except the fact that it does both. It commits mistakes AND it also comes up with remarkable, novel output that makes sense. Perhaps you just haven't seen an example of it that's convincing enough.

https://www.engraved.blog/building-a-virtual-machine-inside/

Here's another one.