[Terr_]

> It's just that since the masses found that they can talk to an AI like a human

In a way it's worse: Even the "talking to" part is an illusion, and unfortunately a lot of technical people have trouble remembering it too.

In truth, the LLM is an idiot-savant which dreams up "fitting" additions to a given document. Some humans have prepared a document which is in the form of a a theater-play or a turn-based chat transcript, with a pre-written character that is often described as a helpful robot. Then the humans launch some code that "acts out" any text that looks like it came from that fictional character, and inserts whatever the real-human-user types as dialogue for the document's human-character.

There's zero reason to believe that the LLM is "recognizing itself" in the story, or that is is choosing to self-insert itself into one of the characters. It's not having a conversation. It's not interacting with the world. It's just coded to Make Document Bigger Somehow.

> they think that it's got human capabilities too

Yeah, we easily confuse the character with the author. If I write an obviously-dumb algorithm which slaps together a story, it's still a dumb algorithm no matter how smart the robot in the story is.

[jdonaldson]

Just wanted to point out that the notion of a "document" is also an illusion to the LLM. It's processing a sequence of low dimensional spaces into another sequence of low dimensional spaces. The input spaces preserve aspects of content similarity based on co-occurrence. The model learns to transform these spaces into higher order spaces based on the outcome of training.

You couldn't say that the model has a singular sense of self, but it certainly has been trained on data that allows it to mimic it in short spurts, and mimicry is what humans do to learn more complex/abstract tasks. The training goal is not to learn how to "be", but rather to learn how to "do" the parts necessary to continue existing.

"Fake it till you make it" is really all that's required to exist in the world.

[physicles]

For humans, the notion of “self” is also an illusion. We’re just atoms dancing to physics. But that’s not the right level of abstraction from which to do psychology.

Is the notion of an LLM operating on a document useful for building an intuition about how the LLM operates? That’s the more interesting question.

[jdonaldson]

I’d say it is, although it’s not really a document but a stream of tokens. The intuition is that the model is able to build abstractions from that stream. Some of the abstractions are related to goal states of the training metric. However, it’s easy to imagine how some of the goal states pertain to the model’s survival in the “wild”, and part of the abstraction that the model captures for this goal is a reflection of itself.

[skrebbel]

> In truth, the LLM is an idiot-savant which dreams up "fitting" additions to a given document.

Tbh I'm not too sure that my brain works fundamentally differently. I'm an idiot-savant who responds to stimuli.

[vrighter]

Do you know the number of atoms in my coffee cup right now?

If you say "No, I don't" then you're already way ahead of any LLM, even the biggest ones. Remember, I asked about how many are in there right now. The only correct answer is "I don't know", because I haven't even told you whether my mug is full or empty.

Here's copilot's answer:

    how many hydrogen atoms are there in my coffee cup right now
    Copilot said:

    That's an interesting question! The number of hydrogen atoms in your coffee cup depends on the volume of coffee and its composition.

    Let's assume you have a standard coffee cup that holds about 240 milliliters (8 ounces) of coffee. Coffee is mostly water, so we can approximate the number of hydrogen atoms based on the water content.

    Water (H₂O) has 2 hydrogen atoms per molecule. The molar mass of water is approximately 18 grams per mole, and 1 mole of water contains Avogadro's number of molecules, which is about (6.022 \times 10^{23}) molecules.

    1 milliliter of water weighs about 1 gram, so 240 milliliters of water weighs 240 grams. This is approximately ( \frac{240}{18} \approx 13.33 ) moles of water.

    Therefore, the number of water molecules in 240 milliliters of water is: [ 13.33 \times 6.022 \times 10^{23} \approx 8.02 \times 10^{24} \text{ molecules} ]

    Since each water molecule has 2 hydrogen atoms, the total number of hydrogen atoms is: [ 2 \times 8.02 \times 10^{24} \approx 1.60 \times 10^{25} \text{ hydrogen atoms} ]

    So, there are approximately (1.60 \times 10^{25}) hydrogen atoms in a standard coffee cup of 240 milliliters. Keep in mind this is a rough estimate, as the actual number can vary based on the exact composition of your coffee.

If I wanted an explanation of how to approximate it, that's what I would have asked for. You could very probably figure that much out.

[lostmsu]

No, this is the correct answer. This is the answer I'd be looking for if I was asking that question. That you failed to specify the question is a failure on your part.

Even stupid models understand that if I ask them the population of Denmark they only need to give rough approximation.

[vrighter]

no, i expect the system to answer the question I asked. Not the question it thinks I wanted to ask. The question is not underspecified, because the point of it is to demonstrate how the llm will never tell you it doesn't know. Nor will it ever tell you your prompt is underspecified.

[lostmsu]

> The question is not underspecified, because the point of it is to demonstrate how the llm will never tell you it doesn't know

I don't see how "is not underspecified" follows from the point you were trying to demonstrate. Yes, you wanted it to be well specified, because otherwise the point doesn't work. But you actually failed to generate such a question cause both people and AI interpret it differently.