[azinman2]

But ultimately it is predicting the next token. That's the taste. Using context from what's already been predicted, what comes before it, attention mechanisms to know how words relate, all of the intermediate embeddings and whatever they signify about the world -- that all just makes the next word prediction that much better.

[oefnak]

But intelligence *is* being able to make predictions! That's the entire reason we evolved intelligence! (Not words, but the world around us, sure, but apparently language makes a pretty good map)

[azinman2]

Prediction is a faction of cognition. There’s a theory of self, perception, sensory fusion, incremental learning, emotions, a world model, communication and a sense of consequences, desire for self preservation and advancement, self-analysis and reflection, goal setting, reward-driven behavior, and so many more aspects that are missing from “predict the next word.”

[Nevermark]

You are confusing the underlying algorithm, such as prediction improved by gradient optimization, with the algorithms that get learned based on that.

Such as all the functional relationships between concepts that end up being modeled, I.e. “understood” and applicable. Those complex relationships are what is learned in order to accomplish the prediction of complex phenomena, like real conversations & text. About every sort of concept or experience that people have.

Deep learning architectures don’t just capture associations, correlations, conditional probabilities, Markov chains, etc. They learn whatever functional relationships that are in the data.

(Technically, neural network style models are considered “universal approximators” and have the ability to model any function given enough parameters, data and computation.)

Your neurons and your mind/knowledge, have exactly the same relationship.

Simple learning algorithms can learn complex algorithms. Saying all they can do is the simple algorithm is very misleading.

It would be like saying logic circuits can only do logic. And’s, Or’s, Not’s. But not realizing that includes the ability to perform every possible algorithm.

[Sharlin]

And how many of those are obvious applications of prediction, where prediction is the hard part?

World model: This is what prediction is based on. That's what models are for.

Sense of consequences: prediction of those consequences, obviously.

Desire for self preservation: prediction; avoiding world states predicted to be detrimental to achieving one's goals.

Goal setting: prediction; predicting which subgoals steer the world towards achieving one's supergoal(s).

Reward-driven behavior: fundamentally interweaved with prediction. Not only is it all about predicting what behaviors are rewarded, the reward or lack thereof is then used to update the agent's model to make better predictions.

There's even a theory of cognition that all motor control is based on prediction: the brain first predicts a desired state of the world, and the nervous system then controls the muscles to fulfill that prediction!