[gwern]

> Humans would say "Left" 100% of the time in a zero-shot scenario as well.

They do not! And you should not just make up assertions like these. You don't know what humans would say. In fact, in polls, they wind up remarkably calibrated. (This is also covered in the cognitive bias literature under 'probability matching'.) People do this poll on Twitter all the time.

[brabel]

Those humans, really difficult to know what they're thinking!

Anyway, humans are fairly predictable when trying to come up with random numbers, for example, have a look at this Veritasium video: https://www.youtube.com/watch?v=d6iQrh2TK98