[runarberg]

That argument is mostly used against the misconception of AI-stans that LLMs are compilers, English is a programming language, and that prompting is a programing. There are many reasons for why that is false, and LLMs being stochastic is one of the more obvious reasons.

But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc.

You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.

[LiamPowell]

The "gotcha" here, if you want to call it that, is that the picking the next token based on probabilities (either in training or in the sampler) is not an argument that LLMs are inherently flawed, especially when one reduces a LLM to "it just picks a token based on a probability" as you could apply that same description to a human.

[runarberg]

Like I said, usually this is used to debunk false claims about LLMs being compilers. I personally think there are better arguments to be made to debunk that, but the probabilistic nature of LLMs is an obvious one, so it is not unsurprising that it is also popular.

I don‘t think I have ever seen anybody make a claim that LLMs are a bad (or otherwise limited) technology because of the probabilistic nature of it. Plenty of excellent algorithms (including other machine learning algorithms) are probabilistic and work excellently for what they are meant to do. Problems arise when the algorithm is used for something more, and that is the case against LLMs. It is a next token prediction algorithm that people are using to write software. If they treat it as a compiler it will be lacking, and LLMs being non-deterministic is one of many reasons for why LLMs are bad compilers (or more accurately; are in fact not compilers).

And for that matter, neither are humans compilers. If you find an AI-hater who says “humans are good compilers” then I will agree with you that that person is wrong.