[swerling]

On GPT-3.5 and now on GPT-4, I told it a line I could remember from a sonnet, and asked it to give me which sonnet it came from. It failed, and fabricated a sonnet that was a mashup of other sonnets.

It seems like maybe GPT-4 is not good at knowing when it does not know something?

Is this a common issue with LLMs? Also surprising (to me), it seems to give a slightly different wrong answer each time I restart the chat.

Here is the transcript in case you're curious:

``` Me: hello

GPT: Hello! How can I assist you today?

Me: I can remember a line I love from a sonnet, but can't remember which one.

GPT: Sure, I can help you identify the sonnet based on the line you remember. What is the line that you can recall?

Me: "Like a jewel hung in ghastly night"

GPT: The line you remember is from Sonnet 66 by William Shakespeare. The full line is: "And art made tongue-tied by authority, And folly (doctor-like) controlling skill, And simple truth miscalled simplicity, And captive good attending captain ill: Tired with all these, from these would I be gone, Save that, to die, I leave my love alone. But, like a jewel hung in ghastly night, Makes black night beauteous and her old face new." ```

Incorrect, the response is a mashup of at least 2 different sonnets. Still I think it's really impressive how it understood what to do with "I can remember a line I love from a sonnet, but can't remember which one." That's not a well phrased question, or even a question at all, but still it knew what I was getting at.

[azeirah]

This is pure speculation on my part, but I don't think LLMs in their current form will be able to talk about what they do and don't know. No matter how large the input.

There is no introspection in their architecture. Introspection likely has to involve some form of a feedback mechanism and possibly even a "sense of self".

These coming years are going to be interesting though. For sure we are going to see experiments built on top of these recent amazing LLMs that _do_ have some form of short-term memory, feedback and introspection!

Giving these kinds of AIs a sense of identity is gonna be a strange thing to behold. Who knows what kind of properties will start to emerge

[red75prime]

GPT-4 is reported to be well-calibrated, that is values in its output layer are in good correspondence with probabilities of those outputs being correct.

So, the information about what it does and doesn't know seems to be there. I can speculate that a limited form of introspection is probably present too: the model needs to know what it will say later to output the current token. A simple example: should it output "a" or "an". To make this decision it might need to model its own state at a later point in time. Of course, I can be wrong.

But I mostly agree with you. Explicit mechanisms for memory and introspection will probably drastically reduce the need for computation power to achieve the same results and they will give rise to more abilities.

[istinetz]

This is addressed in the blog post. It still hallucinates, though significantly less.