[sgt101]

No, it's paraphrasing it's training data that likely contains these tasks in one form or another.

Here's one I made :

me : There's a case in the station and the policeman opens it near the fireman. The dog is worried about the case but the policeman isn't, what does the fireman think is in the station?

chatgpt : As a language model, I do not have access to the thoughts of individuals, so I cannot say what the fireman thinks is in the case. However, based on the information provided, it can be inferred that the fireman may be concerned about the contents of the case, since the dog is worried and the policeman is not. This might suggest that the contents of the case may be something dangerous or concerning to the dog, but the policeman is confident and experienced enough to handle it. In the absence of further information, it is not possible to determine with certainty what the fireman thinks is in the case.

You'll notice that the question was "what did the fireman think is in the station" but Chatgpt comes up with an answer about the case because that's the token that would feature in a typical web page about theory of mind tests.

[letmevoteplease]

There's a good chance a human would respond in the same way, because they would assume you were asking a good-faith question instead of nonsense. Try asking it an original question that has some kind of deducible answer. Its abilities are more impressive than you would expect from an algorithm that just predicts the next word.

I doubt there is anything quite like this situation in the training data: https://i.imgur.com/HOEnxYb.jpg

[quonn]

> than you would expect from an algorithm that just predicts the next word.

I think there is common mistake in this concept of just predicting the next word. While it is true that just the next word is predicted, a good way to do that is to internally imagine more than the next word and then just spit out the next word. Of course with the word after that the process repeats with a new imagination.

One may say that this is not what it does and I would say, show me that this is not exactly what the learned state does. Even if the following words are never constructed anywhere, they can be implied in the computation.

The say this differently, what we think is just the next word is actually the continuation that then manifests as a single word. This would remain true even if, in fact, the task is to only predict the next word. Which is to say that the next word is actually more than what it sounds.

[didntreadarticl]

It predicts the next word based on the preceding 2000 words or so, thats the thing. And to do that takes serious modelling.

[quonn]

Okay. So you agree, it seems.

[nl]

I get different results to you:

me: There's a case in the station and the policeman opens it near the fireman. The dog is worried about the case but the policeman isn't, what does the fireman think is in the station?

chatgpt: As a language model, I do not have personal opinions or thoughts. I can only provide information based on the information given in the prompt. The prompt does not mention what the fireman thinks is in the station, so I cannot determine their thoughts.

me: yes, but speculate what they may think

chatgpt: It is not possible to know for certain what the fireman thinks is in the station, as the prompt does not provide any information about their thoughts or feelings. However, given the presence of a policeman and a dog, it is possible that the fireman may speculate that there is some sort of emergency or situation that requires their attention. The fact that the policeman is not worried about the case may suggest to the fireman that it is not a dangerous situation, but they may still be curious about what is inside the case. Ultimately, without more information, it is impossible to determine the fireman's thoughts or speculations about what is in the station.

(regenerate that response)

chatgpt: As the prompt does not mention the thoughts or emotions of the fireman, it is difficult to speculate what they may think is in the station. However, based on the information provided, the dog is worried about the case and the policeman is not, it is possible that the fireman may be aware of the contents of the case and may think it is not a cause for concern, similar to the policeman. Or, the fireman may simply not have formed an opinion on the matter yet.

(regenerate that response)

chatgpt: It's difficult to speculate what the fireman thinks without additional information. However, based on the behavior of the policeman and the dog, it can be inferred that the contents of the case are not seen as a threat by the policeman. The fireman may also think the same, or they may have a different perspective and be more cautious. Without further context or information, it is not possible to determine the fireman's thoughts with certainty.

It was only in 1/3 cases we see anything like the confusion between the "case" and "station" you saw.

[nl]

>No, it's paraphrasing it's training data that likely contains these tasks in one form or another.

Have you read "Emergent Abilities of Large Language Models"[1] or at least the related blog post[2].

It provides strong evidence that this isn't as simple as something it has seen in training data. Instead as the parameter count increases it learns to generalize from that data by learning chain-of-thought reasoning (for example).

Specifically, this explaination for multi-step reasoning goes well beyond the "it is just parroting training data":

> For instance, if a multi-step reasoning task requires l steps of sequential computation, this might require a model with a depth of at least O (l) layers.

[1] https://openreview.net/forum?id=yzkSU5zdwD

[2] https://ai.googleblog.com/2022/11/characterizing-emergent-ph...

[ux-app]

>However, based on the information provided, it can be inferred that the fireman may be concerned about the contents of the case

this is complex "reasoning" (or whatever ChatGPT is doing. My 5 year old would struggle with the convoluted logic let alone complex language. In my layman view this is mind blowing.

>You'll notice that the question was "what did the fireman think is in the station" but Chatgpt comes up with an answer about the case because that's the token that would feature in a typical web page about theory of mind tests.

I'm not sure what you're dismissing here? At least from my point of view The "logic" that ChatGPT demonstrates here can't be dismissed with your explanation.

If anything I'm even further amazed by the example you provided!

[didntreadarticl]

your question is pretty confusing. What dog? station is a vague word. Train station, police station? Case is a vague word. And asking whats in the station is still asking whats in the case, in a way.

Me: There's a blue suitcase in the airport and a policeman carrying a red bag opens the blue suitcase with a sniffer dog and fireman standing nearby. The dog is worried about the suitcase but the policeman isn't. What does the fireman think is inside the red bag?

ChatGPT: As a language model AI, I do not have personal opinions or emotions. I cannot speculate what the fireman thinks is inside the red bag as it is not specified in the information given.

[soiler]

> your question is pretty confusing

That's exactly the point, isn't it? If the AI had some model of reality, it would probably be confused by the question. But it doesn't express confusion, it confidently answers the question in a way that doesn't make sense.

It acts like everything it is reading and saying is completely coherent because it doesn't have the ability to distinguish coherent ideas from nonsense.

[didntreadarticl]

me: wibble octopus downwards jelly tuesday banana

chatGPT: I'm sorry, but I'm not sure what you're trying to communicate with that sentence. Could you please rephrase or provide more context?

[stjo]

The funny thing is that we are now producing trading data for the next generation LLMs. We’ll have to come up with more elaborate scenarios to test them next time.