[hkalbasi]

> A LLM (...) merely produces syntactically likely combinations of the training data

While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.

[aabhay]

I think the questionable term is not “syntactically likely” but “merely”. Syntactical likeness is a vast solution space that encompasses the work of a terrible developer and a genius developer. In fact this solution space is a gap wide enough to encompass all coding knowledge and expertise.

[rcxdude]

"Combinations of the training data" is also hiding a pretty common and serious misconception about modern generative AI: this seems to come from a mental model that LLMs and image generators are merely producing a collage of sorts, especially that this is some way linked to specific parts of the training data as opposed to a generalized result of building up an understanding of how to produce a sentence, piece of code, or plausible image.

[inigyou]

That is how they feel when using them for code. It feels like they generate templates filled in with the details of a specific case.

[LiamPowell]

The natural next argument that I see a lot is "well it's still all probabilistic", which is technically true. However, don't the atoms that make up the cells that make up a human move around and interact based on probabilities?

Saying a LLM is all just based on probabilities is pretty meaningless, even if it is true, since everything else is just based on probabilities too. What matters is the massive amount of machinery that generates those probabilities. Anything from rolling a dice to an accurate model of an entire human, or even a model of the entire universe, is all "just probabilities".

[runarberg]

That argument is mostly used against the misconception of AI-stans that LLMs are compilers, English is a programming language, and that prompting is a programing. There are many reasons for why that is false, and LLMs being stochastic is one of the more obvious reasons.

But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc.

You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.

[LiamPowell]

The "gotcha" here, if you want to call it that, is that the picking the next token based on probabilities (either in training or in the sampler) is not an argument that LLMs are inherently flawed, especially when one reduces a LLM to "it just picks a token based on a probability" as you could apply that same description to a human.

[runarberg]

Like I said, usually this is used to debunk false claims about LLMs being compilers. I personally think there are better arguments to be made to debunk that, but the probabilistic nature of LLMs is an obvious one, so it is not unsurprising that it is also popular.

I don‘t think I have ever seen anybody make a claim that LLMs are a bad (or otherwise limited) technology because of the probabilistic nature of it. Plenty of excellent algorithms (including other machine learning algorithms) are probabilistic and work excellently for what they are meant to do. Problems arise when the algorithm is used for something more, and that is the case against LLMs. It is a next token prediction algorithm that people are using to write software. If they treat it as a compiler it will be lacking, and LLMs being non-deterministic is one of many reasons for why LLMs are bad compilers (or more accurately; are in fact not compilers).

And for that matter, neither are humans compilers. If you find an AI-hater who says “humans are good compilers” then I will agree with you that that person is wrong.

[runarberg]

I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the training data (i.e. supervised learning) then the original assumption still holds. So I fail to see how this is a misconception.

[atherton94027]

I think OP might be referring to synthetic data, which is now used to provide more data for training?

[runarberg]

That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better.

However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.

[tulio_ribeiro]

Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.

[runarberg]

I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data).

Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.

[xdavidliu]

that's not how the terminology is typical used, even if it makes literal sense

[e12e]

Doesn't the LLM still output the next most probable token (modulus heat/desired variation) according to its model? Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?

[bob001]

So do humans more or less? The only counterpoints I've heard are religious or unmeasured quantum brain something.

[e12e]

I think there are quite fundamental differences to the LLM model equivalent to an adult human mind and current LLMs (even accepting an LLM can model a human brain, which I don't think is a given - but I also don't think it's impossible).

The details matter (training set, full context). I doubt our current LLMs knows what smelling cut grass on a wet morning feels like, or how the stomach flutters on a first kiss - even if they have "read" tens to hundreds of attempts at describing such things.

You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.

[bob001]

>The details matter (training set, full context).

But you said:

>Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?

So which is it?

>You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.

I'm merely noting that your argument that "training+input=>output is the problem" applies to humans. There's other arguments for and against LLMs however I'm simply pointing out that your argument isn't a good one.

[e12e]

> I'm merely noting that your argument that "training+input=>output is the problem" applies to humans.

I'm not sure I ever said that it was a problem.

I also never meant to say that sampling across the model (formed by training) based on input couldn't produce novel token sequences.

As for human vs LLM - my argument would be that while trained on a large corpus, the models are poor in experience and "sensory input" - their training so different from growing up - that even if we could model humans as LLMs - the gap between current LLMs and "human" LLMs remain a wide one.

More to the point - I believe the current generation of LLMs are way too sensitive to input (prompt, system prompt, context/RAG) - precisely because they're still too narrowly trained, and also aligned to weigh input heavily.

What we have today are models that work well with language (generate, transform, translate) - not so great on complex tasks - because they're still generating next probable token.

[bigstrat2003]

No. Humans actually understand things and can reason from one fact to another. We do not just spit out the next most likely thing without any intelligence the way an LLM does.

[josu]

The combination of the most probable tokens doesn't necessarily have to be in the training set, thus creating something completely novel.

[e12e]

If you shuffle a deck of cards, you likely have a novel sequence of cards. I'm not sure how that is relevant?

[josu]

I was responding to this "merely produces syntactically likely combinations of the training data".

[DiabloD3]

That is called a hallucination.

[josu]

Yes, like the Jacobian counterexample.

[inigyou]

That LLM was attached to a Python and MATLAB evaluator. It generated plausible things to feed into a real computer.

[pixl97]

Weird take as a lot of things humans invent and test are done in similar ways.

[Gander5739]

Only if it's incorrect.

[semiquaver]

It’s not sampling anything though. At each inference step the fundamental output of the process can be interpreted as a probability for each possible token how likely it is. Sampling would imply otherwise but the probabilities are the output.

[probably_wrong]

I would like to offer this comment from three days ago as counterpoint:

https://news.ycombinator.com/item?id=49013694

[lifthrasiir]

Common sense testing only tells about the presence of common sense, that's it. It is clear from recent events that LLMs do definitely possess some sort of intelligence even though it might have been polarized to certain domains.

[scotty79]

What does that prove? That AIs are not RLd for counting letters in texts? Do you doubt that they could excel at it with only synthetic data?