[dgellow]

The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models

[simonw]

Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely.

Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.

[wren6991]

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.

[thesz]

  > The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.

https://arxiv.org/abs/2504.13837

"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "

[adw]

This paper is less dramatic than you think it is and really just re-explains what RLVR does.

Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier.

What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.

[wren6991]

Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing.

> Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model

On the face of it this seems unsurprising given the policy gradient term directly minimises this difference.

I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path.

The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model.

Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model. That also aligns with the pass-at-high-k: yes it's doing more exploration without training but it's a bit of a monkeys-on-typewriters situation.

Lack of novelty is readily explained by the fact that you need some nonzero pass rate in the base model to actually get some useful training signal from RLVR. That's a limitation of contemporary RLVR techniques, not a limitation on post-training in general.

I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.

[adw]

Other post-train mechanisms have meaningful bandwidth for introducing new information to the model (on-policy distillation is more or less the other extreme). RLVR aligns the model to particular behaviors it could already (unreliably, intermittently) express, it doesn't introduce new behaviors; the behaviors were latent in the pretrain/midtrain.

[CamperBob2]

VibeThinker 3B is highly problematic for the thesis expressed in that paper. There is no possible way it could derive its reasoning capabilities from a bag of 3B parameters alone.

[simonw]

Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one.

You can get into RL as part of explaining why it's so unnervingly good at picking a next token.

[wren6991]

That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing.

I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834):

> The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e

...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.

[didibus]

> which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.

Isn't G a learned probability?

[wren6991]

I don't think so. Probability of what?

If you put something through a softmax the output is (trivially) a valid PMF. Does that matter? You're not sampling from it.

[didibus]

> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different

I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?

[IanCal]

IMO that doesn’t sound so much like prediction any more.

It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”.

But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.

[didibus]

> predict what would come next in this text sampled from distribution X

It tries to learn the distribution of "useful" results either through verified rewards or human feedback. Then it encodes that in the network. When you run inference later, it samples or selects from that distribution.

Maybe it's a matter of interpretation. It's not predicting the next token based purely on the training corpus's distribution anymore, the RL process fine tunes that distribution so it predicts the next token that is closer to what was rewarded during RL. But as I see it, it's still predicting the next token, just from a reenforcement learned distribution instead of one found in a corpus of data.

[ruszki]

Just because you change loss function or dataset, it doesn’t become not a prediction. Just a prediction of something else.

[IanCal]

What’s it a prediction of?

[slopinthebag]

I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.

[ameliaquining]

I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.

[simonw]

I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text.

It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.

[CamperBob2]

It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.

The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged.

The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"

[amenhotep]

LLMs self-evidently have no experience when not inferencing, seems like the most consequential difference to me. If we are next token predictors then we are next token predictors that are inferencing at every waking moment and arguably much of our sleeping moments too; when we train an LLM that can inference as much as we do and remain coherent then I'll be more worried about whether it might have experience.

[ameliaquining]

I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.)

I recommend this explanation: https://www.astralcodexten.com/p/next-token-predictor-is-an-...

[dgellow]

You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)

[ameliaquining]

Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture?

(Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)

[cyanydeez]

I think it's important to understand the humans _can_ do what LLMs do: predict next tokens from prior ones.

But LLMs are only operating on text and humans are only operating on <waves hands>

[dgellow]

While you’re active in this thread, I just want to say thank you for all your writing, you’re such a reliable source of sanity in that crazy new world :)

[Forgeties79]

> It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.

At the risk of sounding overly flippant, all world domination has been achieved by some person(s) figuring out which word should come next. Words quite literally = action when it comes to LLM’s with tools access

[KPGv2]

> At the risk of seeming X, statement that overwhelmingly demonstrates X-ness.

It's exhausting to even consider where to begin addressing the assertion that good leadership is just predicting the next word to say. Especially considering the corpus available to most great leaders in history was extremely small. To think Hannibal's military campaigns were just because he'd read like ten books in his life and could accurately forecast effective rhetoric is...indescribably divorced from reality.

At the risk of seeming like a jerk.

[notathrowaway51]

"Competent strategy" is just as good a target for a word predictor to optimize as "effective rhetoric" is.

[argee]

Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.

[ameliaquining]

Isn't the outcomes question the one that people actually care about in most contexts?

[argee]

Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).

[ameliaquining]

Sorry, what contexts are these?

[notahacker]

I don't know about "most", but there are a lot of people treating it as somewhere between "magic oracle" and "new friend"

Not to mention people more worried about whether the AI is motivated to hurt us than what human motivations can do with something that can autocomplete its way through every possible attack vector of cryptographic systems most of use would prefer remain secure.

(tbf I think the "glorified autocomplete" still works surprisingly well for programming outcomes too. Autocomplete [and fuzzy search of reference material] actually is useful and often right and certainly can save time even when it's only suggesting the rest of the variable name. But you might not want to commit everything it suggests...)

[svieira]

My favorite analogy for LLMs is "an intellectual chain saw". It does one thing, well, but that isn't "cut wood". It's "cut". What it cuts will happily include limbs.

[ameliaquining]

Whether an AI's operators lose control of it is an outcomes question.

[argee]

Sorry, this discussion is too incommensurable for me.

[red75prime]

Capability or outcome...or internal structure.

What does "glorified autocomplete" say in an ontological sense exactly? Nah. It's just a lazy dismissal.

BTW, autoregressive pretraining (autocomplete) is a part of training.

[inigyou]

But... These are also literally glorified autocomplete. Autoregressive language models are all autocomplete. It turns out that advanced enough autocomplete can do interesting stuff.

[Groxx]

I generally like it still. It describes their failure modes pretty well, and in a way that most people already recognize. They're incomparably more complex, of course, but they are not intelligent and they are very much repeating what they've seen without any capability for factual accuracy.

Practically every other attempt at describing them leans too technical and unfamiliar (stochastic parrot) or too anthropomorphic (even describing them as "not like a human" gets people thinking in terms of humans, like how if I mention that your tongue is in your mouth all the time, using up almost all of the room, feeling your teeth and tasting itself, you're now uncomfortably aware of it and the numerous bumps on the surface).

You need to work from a reference that has both a shared understanding, and does not lead to problematic "if X has Y, and Z is like X, then Z has Y" seemingly-logical derived beliefs. "Spicy autocomplete" is a fairly safe starting point in both ways.

[red75prime]

"Repeating what they've seen" could be, depending on their generalization abilities, something like low-n-gram Markov chain, repeating of surface-level speech patterns, repeating reasoning patterns, repeating mind-theory-level patterns, repeating self-correction patterns and so on.

[Groxx]

And they fall somewhere between "surface-level speech patterns" and "repeating reasoning patterns" imo, yeah. They do indeed learn higher-order patterns than a Markov chain... but they'll blatantly misapply reasoning patterns and self-contradict confidently, just like you see with human bullshitters who don't understand one bit of what they're doing.

They sound fairly human, until you notice the patterns. They sound like they're thinking, until you pay attention.

[gwd]

There was an interview I watched with Cory Doctorow just a few weeks ago, where someone asked him, "What kind of evidence could be presented to you to make you believe that LLMs were now ready to replace people at jobs entirely?" And he basically responded that he didn't think it was possible they ever would, because they're not really thinking, they just "predict the next token". This in spite of him actually having first-hand experience with people getting loads of value out of Claude Code and GPT Codex.

[brookst]

Doctorow is one of those people who makes money from what he believes, so it’s very unlikely he’ll ever change his beliefs.

[bawolff]

Which obviously raises the question of if we are just glorified autocomplete (or glorified pattern matching). After all, we evolved from things just trying to recognize the patterns for what will hurt us and what will cause pleasure.

[brookst]

It’s such willful blindness though. Why deny that there is a huge gulf between autocomplete and sentience? It’s like saying cars are glorified bicycles because they’re sure not airplanes.

[cyanydeez]

I like glorified copy/paste frankensteined with find/replace.