[BugsJustFindMe]

> > It's pretty hard to unscramble as a human.

> Is it? With essentially no knowledge of golf and no lookups, in less than a minute of work, I get

A MINUTE! FOR 23 WORDS! Yes, the fact that you measured in units of the nearest minute for something so short is the sign of it being hard. Compare how long it takes you to read the unscrambled version.

[checkyoursudo]

A language model being able to do something that rarely has to be done in the first place within milliseconds or less that would take a human a minute to do just does not seem that impressive. (I tried it without reading any of these comments below, and I got everything except "clinching" in about 20 seconds.)

I don't know. I am having a hard time overcoming the likelihood that scrambled and cipher-encoded words/solutions are part of the training corpus, thus fully explaining the phenomenon.

If someone can get it to decipher something like the zodiac killer's cipher, then I might be more impressed.

[BugsJustFindMe]

That's ok. You get to be impressed or not impressed by whatever impresses or doesn't impress you. Maybe nothing is impressive! I'm extremely fine with a statement like that.

I question whether your bounding the human time to a minute is valuable here though. If the jumbled content were multiple pages long instead of only 23 words, would it be somehow more impressive despite the process being exactly the same?

> I am having a hard time overcoming the likelihood that scrambled and cipher-encoded words/solutions are part of the training corpus, thus fully explaining the phenomenon.

Scrambled words are part of my training corpus too, but it still takes me a lot longer than the machine, and I don't even need to give the machine a hint about what's going on. I just say "tell me what this says" and a moment later it does.

[thaumasiotes]

I measured in units of minutes because the largest unit below that is the second, which is too small. How accurately do you think people can measure how long it takes them to do mental work?

I would usually determine whether to call something "hard" by reference to a measure of difficulty such as rate of success, not by whether doing it with no practice is slower than doing a similar task that I've practiced extensively.

[BugsJustFindMe]

> I would usually determine whether to call something "hard" by reference to a measure of difficulty

How long it takes you to do something compared to something else is a measure of its difficulty, all else being equal.

> such as rate of success, not by whether doing it with no practice is slower

Then I guess nothing is hard if you can ever eventually succeed, even if you struggle along the way, which sounds to me like not a very useful distinction.

Because that's what you're describing here. You're rapidly failing to interpret each scrambled word as its unscrambled form. You're sampling letters, failing, and trying again, over and over, until you eventually succeed, and then moving on to the next word. Maybe you're even backtracking to previous words that you got wrong (now/won perhaps) based on later unscramblings. And you're ignoring that part and only evaluating the very final outcome in a binary "got to the marathon finish line" fashion while ignoring the shortness of breath and stitch in your side.

The entire reason it takes longer is because you have a low rate of intermediate success, which makes progress slow, even though you got there in the end.

> I measured in units of minutes because the largest unit below that is the second

"Seconds" is an extremely common descriptor for how long something might take. But you didn't say "in seconds". This arbitrary rule about whole units sounds defensive. It's really ok for us to acknowledge the significance of the fact that reading the scrambled version takes significantly more mental effort.

[beepbooptheory]

Could you explain what your issue is here? I think we are generally just try to reason through this phenomena, not make grand conclusions about the model. We talk about how hard/time consuming it is for a human to pose possible theories for what the LLM could be doing. It is not to assert anything about how "difficult" it is for the LLM compared to a human, because we can't ascribe difficulty/ease to anything the model "does", simply because we know the fundamental mechanics of its inferencing. We can only after the fact of an LLM's output say something like: "Wow that would have been hard for me to output" or "I could have written something similar in like 5 minutes." But these claims can only ever be counter factuals like this, because in reality the output of an llm comes out at a constant rate no matter what you prompt it with.

If you try to say more, you'll end up falling in weird contradictions: it would take an llm a lot longer to output 10 million 'a's than a human, so it must be "harder" for the llm to do that than a human.

[bravetraveler]

Don't forget to compare reading and digesting to make it fair! Speed reading doesn't count /s

Apparently 'reading' (glancing at) something a few times is difficult work.

I can give you a bottle of water if you're tuckered out, I didn't need mine.