[lsy]

The difference is that "the world" is not exhaustible in the same way as Go is. While it's surely true that the number of possible overall Go game states is extremely large, the game itself is trivially representable as a set of legal moves and rules. The "world model" of the Go board is actually just already exhaustive and finite, and the computer's work in playing against itself is to generate more varied data within that model rather than to develop that model itself. We know that when Alpha Zero plays a game against itself it is valuable data because it is a legitimate game which most likely represents a new situation it hasn't seen before and thus expands its capacity.

For an LLM, this is not even close to being the case. The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. And we certainly haven't figured out a "model" which would let a computer generate new and valid data that expands its understanding of the world beyond its inputs, so self-training is a non-starter for LLMs. What the LLM is "understanding", and what it is reinforced to "understand" is not the world but the format of texts, and while it may get very good at understanding the format of texts, that isn't equivalent to an understanding of the world.

[famouswaffles]

>The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility.

No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts.

To the bird who can intuitively sense and use electromagnetic waves for motion and guidance, your model of the world is fundamentally incomplete.

There is a projection of the world in text. Moreover training on additional modalities is trivial for a transformer. That's all that matters.

[lsy]

That's the difference though. I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. To a neural network there is no distinction. The closest the entire dynamic comes is the very basic annotation of RLHF which itself is done by an external human who is providing the value judgment, but even that is absent once training is over.

Despite not having the bird's sense for electromagnetic waves, I have an understanding that they are there, because humans saw behavior they couldn't describe and investigated, in a back-and-forth with a world that has some capacity to disprove hypotheses.

Additional modalities are really just reducible to more kinds of text. That still doesn't exhaust the world, and unless a machine has some ability to integrate new data in real time alongside a meaningful commitment and accountability to the world as a world, it won't be able to cope with the real world in a way that would constitute genuine intelligence.

[famouswaffles]

>I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins.

Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception.

And yes LLMs can in fact tell truth from fiction.

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

Your argument seems to boil down to "they can't perform experiments" but that isn't true either.

[user_named]

It is a very basic fact that LLMs have no concept of true or false, it only has an ability to look up what text data it has seen before. If you do not understand this you are in no position to discuss LLMs.

[SpicyLemonZest]

I really don't know what people mean when they say this. We routinely instruct computer chips to evaluate whether some condition is true and take action on that basis, even though the chip is "just" a selectively doped rock. Why would the details of an LLM's underlying architecture mean that it can't have a concept of true or false?

[xcv123]

One of the most ridiculous comments I have read about LLMs here.

The ~100 layer deep neural networks infer many levels of features over the text, including the concept of true and false. That is trivial for an LLM.

Are you completely unaware these are based on deep neural networks?

Convolutional Neural Networks don't operate by "look up" of text data.

[user_named]

Okay, so then tell me how does it decide whether it is true or false that Biden is the POTUS?

It's response is not based on facts about the world as it exists, but on the text data it has been trained on. As such, it is not able to determine true or false even if the response in the above example would be correct.

[robbrown451]

It certainly doesn't "look up" text data it has seen before. That shows a fundamental misunderstanding of how this stuff works. That's exactly why I use the example above of Alpha Zero and how it learns to play Go, since that demonstrates very clearly that it's not just looking things up.

And I have no idea what you mean by saying that it has no concept of true or false. Even the simplest computer programs have a concept of true or false, that's kind of the simplest data type, a boolean. Large language models have a much more sophisticated concept of true and false that has a lot more nuance. That's really a pretty ridiculous thing to say.

[user_named]

Yes, you don't understand what I said. The model has no concept of true or false. It only has embeddings. If 'asked' a question it can see if that is consistent with its embeddings and probabilities or not. This is not a representation of the real world, of facts, but simply a product of its training.

[jeffparsons]

They have no inherent concept of true or false, sure. But what are you comparing them to? It would be bold to propose that humans have some inherent concept of true or false in a way that LLMs do not; for both humans and LLMs it seems to be emergent.

[mattigames]

In all these arguments its implied that this "genuine intelligence" is something humans all have, and nothing could be farther from the truth, that is why we have flat earthers or religious people and many other people beliving for decades easily refutable lies.

[astrange]

There is no such thing as a world model, and you don't have one of them. This is a leftover bad psychological concept from the 70s AI researchers who never got anywhere. People and other creatures do very little modeling things, they mostly just do stuff.

[xcv123]

World model means inner representation of the external world. Any organism with a functioning brain has a world model. That's what brains do.

If you don't have a world model then you are a vegetable and could not be replying on HN.

[astrange]

If you close your eyes, how long can you navigate in the environment without hitting something? Not long, because you didn't model it.

If you're taking out the recycling, do you take the time to identify (model) each piece of it first? No, because that's not necessary.

[xcv123]

Wait, you actually think we are talking about modelling as a conscious deliberate process in active working memory? Well there's your fundamental mistake. That is not what we are discussing, not even remotely.

The vast model in your brain is learned and generated unconsciously without your direct awareness.

[jofla_net]

I do agree, but more importantly love this part of the argument! Its when all the personality differences become too much to bear and suddenly people are accused of not even knowing themselves. Been there before, what a wild ride!

[Dylan16807]

> suddenly people are accused of not even knowing themselves

It's not some desperate retort. People don't know themselves very well. Look at the research into confabulation, it seems to be standard operating procedure for human brains.

[pizza]

Kant would like a word with you about your point on whether people themselves understand the world and not just the format of their perceptions... :)

I think if you're going to be strict about this, you have to defend against the point of view that the same 'ding an sich' problem applies to both LLMs and people. And also whether if you had a limit sequence of KL divergences, one from a person's POV of the world, and one from an LLM's POV of texts, what it is about how a person approaches better grasp of reality - and likewise their KL divergence approaches 0, in some sense implying that their world model is becoming the same as the distribution of the world - that can only apply to people.

It seems possible to me that there is probably a great deal of lurking anthropocentrism that humanity is going to start noticing more and more in ourselves in the coming years, probably in both the direction of AI and the direction of other animals as we start to understand both better

[tazjin]

The world on our plane of existence absolutely is exhaustible, just on a much, much larger scale. Doesn't mean that the process is fundamentally different, and for the human perspective there might be diminishing returns.

[kubiton]

What if we are just the result of a ml network with a model of the world?

[user_named]

We're not.