[RandomLensman]

Defining intelligence as prediction leaves out a lot of other things that humans would see as intelligence in other humans (e.g., creating a novel), also quite simple organisms make predictions (e.g., a predator jumping at prey makes a prediction about positions).

[coldtea]

>Defining intelligence as prediction leaves out a lot of other things that humans would see as intelligence in other humans (e.g., creating a novel)

Would it?

Why would "creating a novel" by a human not itself be text generation based on prediction on what are the next good choices (of themes, words, etc) based on a training data set of lived experience stream and reading other literature?

[RandomLensman]

What is the human predicting there? Why would it need to be a prediction task at all? How about a dada-ist poem? Made-up words and syntax? If it is prediction but the criterion for "what is a good next choice" can totally be made up on the fly - what does the word "prediction" even mean?

[coldtea]

>What is the human predicting there?

Their next action - word put on page, and so on.

>Why would it need to be a prediction task at all?

What else would it be?

Note that prediction in LLM terminology doesn't mean "what is going to happen in the future" like Nostradamus. It means "what is a good next word given the input I was given and the words I've answered so far".

>How about a dada-ist poem? Made-up words and syntax?

How about it? People have their training (sensory input, stuff they're read, school, discussions) and sit to predict (come up with, based on what they know) a made-up word and then another.

[RandomLensman]

That is a meaningless definition of prediction if "what is a good next word" has an ever changing definition in humans (as everything would fulfill that definition).

[coldtea]

That's the very definition of production in an LLM.

What does "has an ever changing definition" mean?

And why "everything would fulfill that definition"?

At any time whats the "good next word" is based on the state created by our inputs thus far (including chemical/physiological state, like decaying memories, and so on). And not only not "everything fullfil it", but it can be only a single specific word.

(Same as if we include the random seed among an LLM output: we get the same results given the same training and same prompt).

[RandomLensman]

"it can be only a single specific word" - that is incorrect as a human can change the process to generate the next word, up to and including, using a random process to create or select the next word (i.e., any word would be fine).

You could say the process chosen is somehow predetermined (even if the choices then are all made by using randomness), but then really the word "prediction" has very little meaning as the criteria to what is a "good next word" have a nearly unlimited and ever changing range as the generating process changes.

[duskwuff]

> Why would "creating a novel" by a human not itself be text generation based on prediction on what are the next good choices (of themes, words, etc) based on a training data set of lived experience stream and reading other literature?

Unless you're Stephen King on a cocaine bender, you don't typically write a novel in a single pass from start to finish. Most authors plan things out, at least to some degree, and go back to edit and rewrite parts of their work before calling it finished.

[astrange]

That can be expressed as text prediction. You output version 1 then output editing instructions or rewritten versions until you're done.

The real issue is running out of the input window.

[anko]

> The real issue is running out of the input window.

isn't this what abstractions are for? you summarise the key concepts into a new input window?

[astrange]

Sure, but if we're talking about editing an entire book eventually the fine details do matter. That, and presumably human authors' abstraction/memories of their books are stored in some more compact form than language tokens. Though we can't be sure about that.

[HarHarVeryFunny]

Maybe a better way to say it rather than "intelligence is prediction" is that prediction is what supports the behaviors we see as intelligent. For example, prediction is the basis of what-if planning (multi-step prediction), prediction (as LLMs have proved) is the basis of leaning and using language, prediction is the basis of modelling other people and their actions, etc. So, ultimately the ability to write a novel, is a result of prediction.

Yes, an insect (a praying mantis, perhaps) catching another is exhibiting some degree of prediction, and per my definition I'd say is exhibiting some (smallish) degree of intelligence in doing so, regardless of this presumably being a hard-coded behavior. Prediction becomes more and more useful the better you are at it, from avoiding predators, to predicting where the food is, etc, so this would appear to be the selection pressure that has evolved our cortex to be a very powerful prediction machine.

[spookie]

I think you're confusing prediction with ratiocination.

I'm sure you've deducted hypothesis' based solely on the assertion that "contradiction and being are incompatible". Note, there wasn't prediction involved on that process.

I consider prediction as a subset of reason, but not the contrary. Therefore, I beg to differ on the whole assumption that "intelligence is prediction". It's more than that, prediction is but a subset of that.

This is perhaps the biggest reason for the high computational costs of LLM's, because they aren't taking the shortcuts necessary to achieve true intelligence, whatever that is.

[HarHarVeryFunny]

> I think you're confusing prediction with ratiocination.

No, exactly not! Prediction is probabalistic and liable to be wrong, with those probabilities needing updating/refining.

Note that I'm primarily talking about prediction as the brain does it - not about LLMs, although LLMs have proved the power of prediction as a (the?) learning mechanism for language. Note though that the words predicted by LLMs are also just probabilities. These probabilities are sampled from (per a selected sampling "temperature" - degree of randomness) to pick which word to actually output.

The way the brain learns, from a starting point of knowing nothing, is to observe and predict that the same will happen next time, which it often will, once you've learnt what observations are appropriate to include or exclude from that prediction. This is all highly probabalistic, which is appropriate given that the thing being predicted (what'll happen if I throw a rock at that tiger?) is often semi-random in nature.

We can better rephrase "intelligence is ability to predict well", as "intelligence derives from ability to predict well". It does of course also depend on experience.

One reason why LLMs are so expensive to train is because they learn in an extremely brute force fashion from the highly redundant and repetitive output of others. Humans don't do that - if we're trying to learn something, or curious about it, we'll do focused experiments such as "Let's see what happens if I do this, since I don't already know", or "If I'm understanding this right, then if I do X then Y should happen".

[RandomLensman]

The ability to write a novel is different from actually writing a novel. If prediction forms the basis of (at least some forms of) intelligence, intelligence itself is more than prediction.

[HarHarVeryFunny]

That's why I say our vocabulary for talking about these things leaves something to be desired - the way we use the word "intelligence" combines both raw/potential ability to do something (prediction), and the experience we have that allows that ability to be utilized. The only way you are going to learn to actually write a novel is by a lot of reading and writing and learning how to write something that provides the experience you hope it to have.

[RandomLensman]

Kind of agree. I think, though, trying to shoe-horn intelligence into some evolutionary concepts is tricky because it is easy stack hypotheses there.

[coldtea]

>The ability to write a novel is different from actually writing a novel

In what way, except as in begging the question?

[RandomLensman]

Which LLM will on its own go and write a novel? Also, even for humans, just because you technically know how to write a novel, you might fail at it.

[coldtea]

>Which LLM will on its own go and write a novel?

Which human will?

We get prompts all the time, it's called sensory input.

Instead of "write a noval" it's more like information about literature, life experience, that partner who broke our heart and triggered our writing this personal novel, and so on.

[RandomLensman]

Some people write novels, some don't. Why some people do so we sometimes know, sometimes we don't (maybe they flipped a coin to decide). Some start to write but fail to finish.

You have to believe that humans have no free will in a certain way to have them be like an LLM, i.e, every action is externally driven and determined.

[jimbokun]

LLMs have shown that writing a novel can be accomplished as an application of prediction, at least to a certain level of quality.

[RandomLensman]

I have yet to see an LLM write a novel on its volition.