[qsera]

>Probable given what?

The training data..

>predicting what intelligence would do

No, it just predict what the next word would be if an intelligent entity translated its thoughts to words. Because it is trained on the text that are written by intelligent entities.

If it was trained on text written by someone who loves to rhyme, you would be getting all rhyming responses.

It imitates the behavior -- in text -- of what ever entity that generated the training data. Here the training data was made by intelligent humans, so we get an imitation of the same.

It is a clever party trick that works often enough.

[empath75]

It is impossible to accurately imitate the action of intelligent beings without being intelligent. To believe otherwise is to believe that intelligence is a vacuous property.

[slopinthebag]

An unintelligent device can accurately imitate the action of intelligent beings within a given scope, in the same way an actor can accurately imitate the action of a fictional character in a given scope (the stage or camera) without actually being that character.

If the idea is that something cannot accurately replicate the entirety of intelligence without being intelligent itself, then perhaps. But that isn't really what people talk about with LLMs given their obvious limitations.

[xlii]

So the actors who portrait great thinkers are great thinkers?

[bonoboTP]

No, actors recite a pre-written script. But scriptwriters do have to be great thinkers in order to know what the great thinker would actually say.

[kqr]

I suppose they really only have to be good at knowing what sort of thing the audience would believe a great thinker would say. As long as the audience does not consist of great thinkers they also cannot know for sure what a great thinker would say.

[bonoboTP]

That's true for unverifiable "talk professions" where there is no grounding and it's all self-referential navel-gazing chatter.

But LLMs are already beyond that in writing code that passes actual tests, proving theorems that are check able with formal methods etc.

The people who still say LLMs are just parrots in 2026 will just keep saying this no matter what, so I don't think it makes sense to argue this point further.

[qsera]

No no, parrots are truly intelligent.

[jeremyjh]

Which is why so many portrayals are unconvincing.

[qsera]

>It is impossible to accurately imitate the action of intelligent beings without being intelligent.

Wait what? So a robot who is accurately copying the actions of an intelligent human, is intelligent?

[empath75]

That was probably phrased poorly. If a robot can independently accurately do what an intelligent person would do when placed in a novel situation, then yes, I would say it is intelligent.

If it's just basically being a puppet, then no. You tell me what claude code is more like, a puppet, or a person?

[qsera]

It is neither puppet or a person. It is a computer program.

[throw310822]

As much as a bundle of an mp3 decoder and a terabyte of mp3 music are "just a program".

[UltraSane]

How can you distinguish intelligence form a sufficiently accurate imitation of intelligence?

[slopinthebag]

By "sufficiently accurate" do you mean identical? Because if so, it's not an imitation of intelligence at all, and the question is thus nonsensical.

[UltraSane]

"it's not an imitation of intelligence at all"

But that is the key insight, how can you tell when an imitation of intelligence becomes the real thing?

[qsera]

When it stops hallucinating without explicit checks for that!

[throw310822]

> The training data

If the prompt is unique, it is not in the training data. True for basically every prompt. So how is this probability calculated?

[cbovis]

The prompt is unique but the tokens aren't.

Type "owejdpowejdojweodmwepiodnoiwendoinw welidn owindoiwendo nwoeidnweoind oiwnedoin" into ChatGPT and the response is "The text you sent appears to be random or corrupted and doesn’t form a clear question." because the prompt doesnt correlate to training data.

[newswasboring]

> The prompt is unique but the tokens aren't.

The tokens aren't unique, but the sequence is. Every input this model sees in unique. Even tokens are not as simple as they seem

If you type "ejst os th xspitsl of fermaby?" in ChatGPT it responds with

> It looks like you typed “ejst os th xspitsl of fermaby?”, which seems like a garbled version of:

> "What is the capital of Germany?”

> The capital of Germany is Berlin.

> If you meant to ask something else, feel free to clarify!"

edit: formatting

[ajam1507]

The prompt does correlate to its training data. In this case, since you sent random text, it generated the most likely response to random text.

[HDThoreaun]

Or because the text you send was random and doesnt form a clear quesiton?

[hmmmmmmmmmmmmmm]

...? what is the response supposed to be here?

[hmmmmmmmmmmmmmm]

Hamiltonian paths and previous work by Donald Knuth is more than likely in the training data.

[red75prime]

The specific sequence of tokens that comprise the Knuth's problem with an answer to it is not in the training data. A naive probability distribution based on counting token sequences that are present in the training data would assign 0 probability to it. The trained network represents extremely non-naive approach to estimating the ground-truth distribution (the distribution that corresponds to what a human brain might have produced).

[qsera]

>the distribution that corresponds to what a human brain might have produced..

But the human brain (or any other intelligent brain) does not work by generating probability distribution of the next word. Even beings that does not have a language can think and act intelligent.

[hmmmmmmmmmmmmmm]

You are always making predictions based on the context. That's why illusions can be so effective like these ones: https://illusionoftheyear.com/cat/top-10-finalists/2024/

[astrange]

LLMs also don't work by generating probability distributions of the next word. Your explanation isn't able to explain why they can generate words, let alone sentences.

[qsera]

That is exactly how they work.

[red75prime]

[Citation needed] Neuroscience isn't yet at a point when it can say this with any certainty.

Anyway. It's not a theorem that you can be intelligent only if you fully imitate biological processes. Like flight can be achieved not only by the flapping wings.

[qsera]

>you can be intelligent only if you fully imitate biological processes

It is not that. It is about having an understanding of how it is trained. For example, if it was trained on ideas, instead of words, then it would be closer to intelligent behavior.

Someone will say that during training it builds ideas and concepts, but that is just a name that we give for the internal representation that results from training and is not actual ideas and concepts. When it learns about the word "car", it does not actually understand it as a concept, but just as a word and how it can relate to other words. This enables it to generate words that include "car" that are consistent, projecting an appearance of intelligence.

It is hard to propose a test for this, because it will become the next target for the AI companies to optimize for, and maybe the next model will pass it.

[hmmmmmmmmmmmmmm]

Obviously there is some level of memorisation involved. That's why you can even get LLMs to write parts of Harry Potter from scratch with perfect precision.

[qsera]

Just using a scaled up and cleverly tweaked version of linear regression analysis...

[red75prime]

That is, the probability distribution that the network should learn is defined by which probability distribution the network has learned. Brilliant!