[kypro]

I disagree. ChatGPT is trained to give responses that humans "like". It's almost certainly learnt that humans don't like to be blatantly lied to, and where possible it's generally best to tell the "truth"[1]. And only if you can't do that the second best option is to tell a convincing lie.

In my experience it rarely ever produces utter BS, but believable BS, which suggests it is at a minimum it's always trying to look like it's telling the truth. And to do this you need to have some concept of truth.

[1] When I say truth I'm referring to what the typical user of ChatGPT believes to be true - not necessarily an objective or provable truth.

[zeven7]

This seems like a good, simple example of complete BS:

> How many words are in this sentence?

> ChatGP: Nine words.

> List the words

> ChatGPT: How, many, words, are, in, this, sentence?

> Please count the words in the list

> ChatGPT: Seven words.

> Is that what you said before?

> ChatGPT: No, I previously said "Nine words." when counting the number of words in the sentence "How many words are in this sentence?"

> Which answer is correct?

> ChatGPT: Both answers are correct. The sentence "How many words are in this sentence?" has nine words, and the list "How, many, words, are, in, this, sentence?" has seven words.

> How can the sentence and the list from the sentence have different numbers of words?

> ChatGPT: The sentence "How many words are in this sentence?" and the list "How, many, words, are, in, this, sentence?" have different numbers of words because the sentence is a complete thought in natural language, while the list is a representation of the individual words that make up the sentence separated by commas. In the sentence, there are nine words connected by grammatical rules to form a coherent message. In the list, the words are separated and listed one by one, so each word is counted as one item in the list.

[Shocka1]

This reminds me of when I asked ChatGPT to construct a simple encryption scheme that increments each letter by one character:

A = B

B = C

...

Z = A

After explicit instructions were given and repeated back to me, I asked it to encrypt several 5 to 10 word sentences, which it consistently screwed up. Not terribly, but one or two words max, with no particular pattern that I could immediately see. I pointed out the issue and tried getting it to adjust to the correct answer, but it kept insisting that it was correct. I gave up soon after.

[amrocha]

Have you considered that you may just not have the expertise necessary to know when you've been fed bullshit?