[jacobgold]

Drawing the conclusion that "humans fail" and "models fail", so they must be similar, is very wrong.

You could have humans calculate 2+2 all day and get a surprisingly high error rate. That reveals a flaw in how humans operate.

LLMs fail for entirely different reasons. Their mistakes don't imply they're human-like at all.

It's not about the error rate.

[ACCount37]

You're saying that a class of mistakes points out a "major (possibly fundamental) flaw". I'm pointing out some very similar classes of mistakes in humans - well known, well documented and widely exploited. They just keep paying the "IRS" in gift cards, buying lottery tickets and getting the captain's age wrong.

If you're using the existence of flaws in LLMs to deny the claim of intelligence to them, then why do "generally intelligent" humans exhibit some impressively similar-looking flaws?

And, if we're talking about that conspicuous similarity - do they actually fail "for entirely different reasons"? Or do you just want the reasons to be "entirely different" - and not the same reasons viewed at a different angle?

Because the similarities between humans falling for trick questions or scams, and LLMs falling for adversarial questions or prompt injections don't look coincidental to me at all.

One of the oldest patterns in scamming is overwhelming and confusing the victim. Numerous prompt injection methods seek to overwhelm and confuse an LLM - if an LLM can't keep track of things, can't grasp what's going on, it's far more likely to lose track of what's a prompt and what's data, overlook past instructions or go past its behavioral guardrails.

And humans who fall for trick questions like "1kg of feathers" or "captain's age" due to shallow attention and naive pattern matching? They fail in surprisingly similar ways to how LLMs fail on SimpleBench tasks that are filled with overwhelming adversarial distractors. Many "trick questions" are tricky to humans and LLMs alike - to the point that it's unlikely to be coincidental.

[jacobgold]

> If you're using the existence of flaws in LLMs to deny the claim of intelligence to them...

That's not the point at all. It's the fact that they fail in ways completely unlike humans.

You also have the burden of proof reversed. Its on you to prove these LLM agents are human-like intelligences if that's your claim. No one can prove this because it's false.

[ACCount37]

You are the one claiming that "they fail in ways completely unlike humans" insistently. Now go cough up some proof. I'll wait.

[Jensson]

If they didn't you wouldn't need the operator, you'd have replaced all your programmers with no drawbacks by now. As long as we keep hiring humans that is all the evidence you need that these AI fails in ways humans don't.

[gghackernewsgg]

Junior developers fail in different ways than senior developers too; that's why seniors oversee juniors. But this doesn't necessarily mean that the senior's and junior's intelligences differ in kind

Your argument "AI needs supervision, therefore it fails in different ways than its operator does" holds.

Your argument "AI fails in different ways than its operator, therefore the AI's intelligence is different in kind" doesn't hold.

[jacobgold]

This isn't even controversial. The proof is available to anyone who uses these systems:

They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.

[vidarh]

I switch languages "randomly" all the time when I think about something in another one of the languages I know. Some word will trigger it and before I know it I will continue in the other language.

In fact just the other day I commented on it to my fiancee after I randomly switched to French because we were discussing a trip and I mentioned a French location and pronounced it in French, and suddenly I was in "French mode" entirely unintentionally and it took a sentence before I realised.

That you think this is unique to LLM's suggests you simply don't know the diversity of human thought as well as perhaps you think you do. That's fine - none of us have a very complete view of that.

[jacobgold]

Yes, people speak multiple languages and switch between them. But it's a superficial analogy to the behavior of LLMs which do something different and for different reasons.

[TeMPOraL]

So just like me, including the prompt injections if you count "nerd sniping" as such?

(And in particular, switching languages on the fly is normal for people who speak more than one well, it's something you learn not to do for the sake of people less comfortable with the languages involved.)

[jacobgold]

> So just like me, including the prompt injections if you count "nerd sniping" as such?

Who would count that as prompt injection? It's a superficial analogy.

If you were vulnerable to prompt injection, I could order you to do absolutely anything you're capable of doing and you would be helpless to do otherwise.

[saberience]

Humans make all these mistakes too, in fact, humans make more mistakes than AI does in coding at the moment.