[dTal]

They predict the "most likely" token given the context. That's a huge caveat. Just putting "this is excellent code" in the context makes a vanilla LLM do better. Doesn't that make you pause for a moment before asserting hard limits on what they're capable of?

You might argue they're still capped at the "best" quality seen in the input. Not so. Take typos. Human text has a certain base rate of typographical errors. LLM output contains almost none. Why? Because there are many more ways to be wrong than right. LLMs are not just averaging machines, they also denoise. That should also give you pause.