[Salgat]

You've missed the point of the example, of course it's not the exact same thing. With regard to LLM, the biggest difference is that it's a regression against the world's knowledge, like an actor who memorized every question that happens to have an answer written down in history. If you give him a novel question, he'll look at similar questions and just hallucinate a mashup of the answers hoping it makes sense, even though he has no idea what he's telling you. That's why LLMs do things like make up nonsensical API calls when writing code that seem right but have no basis in reality. It has no idea what it's doing, it's just trying to regress code in its knowledge base to match your query.

[retsibsi]

I don't think I missed the point; my point is that LLMs do something more complex and far more effective than memorise->regurgitate, and so the original analogy doesn't shed any light. This actor has read billions of plays and learned many of the underlying patterns, which allows him to come up with novel and (often) sensible responses when he is forced to improvise.

[captain_coffee]

> LLMs do something more complex and far more effective than memorise-regurgitate

They literally do not, what are you talking about?

[esafak]

What kind of training data do you suppose contains an answer to "how to build a submarine out of spaghetti on Mars" ? What do you think memorization means?

https://chatgpt.com/s/t_6942e03a42b481919092d4751e3d808e