[jacobgold]

It's one example that points out a major (possibly fundamental) flaw. I can point to prompt injection as another example. There are tons more if you're interested.

Are you actually claiming LLMs operate based on human-like intelligence?

[ACCount37]

We're on Hacker News. Do I really have to point out the existence of social engineering to you? Or that scamming old people out of their life savings is a profitable enough activity that there are entire call centers dedicated to the task?

Humans keep overestimating just how high the bar of "human-like intelligence" is.

[jacobgold]

Drawing the conclusion that "humans fail" and "models fail", so they must be similar, is very wrong.

You could have humans calculate 2+2 all day and get a surprisingly high error rate. That reveals a flaw in how humans operate.

LLMs fail for entirely different reasons. Their mistakes don't imply they're human-like at all.

It's not about the error rate.

[ACCount37]

You're saying that a class of mistakes points out a "major (possibly fundamental) flaw". I'm pointing out some very similar classes of mistakes in humans - well known, well documented and widely exploited. They just keep paying the "IRS" in gift cards, buying lottery tickets and getting the captain's age wrong.

If you're using the existence of flaws in LLMs to deny the claim of intelligence to them, then why do "generally intelligent" humans exhibit some impressively similar-looking flaws?

And, if we're talking about that conspicuous similarity - do they actually fail "for entirely different reasons"? Or do you just want the reasons to be "entirely different" - and not the same reasons viewed at a different angle?

Because the similarities between humans falling for trick questions or scams, and LLMs falling for adversarial questions or prompt injections don't look coincidental to me at all.

One of the oldest patterns in scamming is overwhelming and confusing the victim. Numerous prompt injection methods seek to overwhelm and confuse an LLM - if an LLM can't keep track of things, can't grasp what's going on, it's far more likely to lose track of what's a prompt and what's data, overlook past instructions or go past its behavioral guardrails.

And humans who fall for trick questions like "1kg of feathers" or "captain's age" due to shallow attention and naive pattern matching? They fail in surprisingly similar ways to how LLMs fail on SimpleBench tasks that are filled with overwhelming adversarial distractors. Many "trick questions" are tricky to humans and LLMs alike - to the point that it's unlikely to be coincidental.

[jacobgold]

> If you're using the existence of flaws in LLMs to deny the claim of intelligence to them...

That's not the point at all. It's the fact that they fail in ways completely unlike humans.

You also have the burden of proof reversed. Its on you to prove these LLM agents are human-like intelligences if that's your claim. No one can prove this because it's false.

[ACCount37]

You are the one claiming that "they fail in ways completely unlike humans" insistently. Now go cough up some proof. I'll wait.

[Jensson]

If they didn't you wouldn't need the operator, you'd have replaced all your programmers with no drawbacks by now. As long as we keep hiring humans that is all the evidence you need that these AI fails in ways humans don't.

[jacobgold]

This isn't even controversial. The proof is available to anyone who uses these systems:

They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.

[Kim_Bruning]

> Are you actually claiming LLMs operate based on human-like intelligence?

Ok, so we've established that it doesn't work like a human being. To paraphrase Dijkstra: The submarine doesn't swim.

But does it exactly sail either? An LLM doesn't exactly work like traditional deterministic software either, does it?

And yet it moves. You can put in data and ask it to process it, and you'll get an answer that's in some ballpark. Closer to quantum or stochastic computing perhaps, but that's not it either, is it? Or SAT-solving? Eh. It's its own computing approach. If you have a problem where the asking is hard but the verification is cheap, it might just be the right tool for the job.