[geysersam]

> Humans would say "Left" 100% of the time in a zero-shot scenario as well.

How can you know what all humans would do?

If the humans interpreted the task correctly, that is, if they understood they will only be asked once, but in a hypothetical repeated experiment the result should still be 80/20, they would certainly not always say "left".

[bena]

Because it's a stupid prompt. Especially for humans.

Because you're really asking what they think the first response would be. That's left. If I knew a machine would pick left 80% of the time, I would bet left 100% of the time. And I'd be right about 80% of the time, which isn't perfect, but is profitable.

[ben_w]

A human brain can't be perfectly reset, the way an AI can.

I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.

[hwillis]

The LLM isn't being perfectly reset. It chooses words randomly; internally it should be slightly different every time. That's the whole point of temperature.

[tempusalaria]

Temperature has nothing to do with internals. Temperature is purely to do with how the logits outputted by the network are transformed into probabilities, which is completely deterministic and not learned. In fact, temperature makes it impossible for LLMs to simulate this kind of probability. As a calibrated 80-20 split at a certain low temperature would be a different split with some other temperature.

[itsgrimetime]

assuming the humans don’t know what the other responses were, I can’t imagine it actually coming out 80/20

[sqeaky]

When polls like these are run the numbers don't always wind up tilted in the favor of the bigger number. I wish I could provide a specific source but I've been listening to the 538 podcast for years and I know they've covered exactly this topic.

Your inability to believe a thing doesn't prevent it from being true.

I would grab a D20 and on a 16 or less I would say left otherwise I would say right. Some people would pick right just because they can. I imagine most people would pick left because it's the 80%. I imagine plenty of people would double and triple guess and waffle then say something.

Few people, even the dumbest among us, are easily modelable deterministic automata.

[bmacho]

Because some humans don't even understand the question. Others do, but they don't know how to solve it.

But other humans can do this task. If you do it with a bunch of mathematicians in a room, you will get your 20-80 distribution.