Even people who _know_ that often seem to have difficulty intuitively believing it, is the trouble; it's very good at _appearing_ to be intelligent, good enough that even people who should know better sometimes think that the correctness problems are just a case of "need more GPUs", rather than insoluble.
Like clockwork, out come the "but humans" deflections. An LLM is not a human-like intelligence. This is patently obvious, such comparisons are nonsensical and just further the problem of people anthropomorphizing a tool and treating it like an oracle.
I did, I said they aren't human-like intelligences, so countering with "humans make mistakes, are humans not intelligent?" is drawing a false equivalence between humans and LLMs.
Since we do not possess a definition of intelligence that isn't human-like, it would be meaningless to argue if LLMs are intelligent in general. All that can be said is that they are not intelligent in the way that humans are.
The question you answered was rhetorical; obviously humans are intelligent. There is another question that actually has an interesting answer after it. In fact, it's not possible to have a meaningful discussion without answering it. I thought that was obvious :)
Maybe I'm miscommunicating, I know that the question about whether or not humans are intelligent is rhetorical. What I'm trying to say is that "humans make mistakes and are obviously intelligent, so a probabilistic syllable generator can be considered intelligent despite making mistakes" does not necessarily follow because at best they're different kinds of intelligences.
In my rush to be a smartass, I did miss that you also asked what definition of intelligence they were working with though, so I suppose I didn't really add anything besides unnecessary snark (-‿-")