[BugsJustFindMe]

> I would usually determine whether to call something "hard" by reference to a measure of difficulty

How long it takes you to do something compared to something else is a measure of its difficulty, all else being equal.

> such as rate of success, not by whether doing it with no practice is slower

Then I guess nothing is hard if you can ever eventually succeed, even if you struggle along the way, which sounds to me like not a very useful distinction.

Because that's what you're describing here. You're rapidly failing to interpret each scrambled word as its unscrambled form. You're sampling letters, failing, and trying again, over and over, until you eventually succeed, and then moving on to the next word. Maybe you're even backtracking to previous words that you got wrong (now/won perhaps) based on later unscramblings. And you're ignoring that part and only evaluating the very final outcome in a binary "got to the marathon finish line" fashion while ignoring the shortness of breath and stitch in your side.

The entire reason it takes longer is because you have a low rate of intermediate success, which makes progress slow, even though you got there in the end.

> I measured in units of minutes because the largest unit below that is the second

"Seconds" is an extremely common descriptor for how long something might take. But you didn't say "in seconds". This arbitrary rule about whole units sounds defensive. It's really ok for us to acknowledge the significance of the fact that reading the scrambled version takes significantly more mental effort.

[beepbooptheory]

Could you explain what your issue is here? I think we are generally just try to reason through this phenomena, not make grand conclusions about the model. We talk about how hard/time consuming it is for a human to pose possible theories for what the LLM could be doing. It is not to assert anything about how "difficult" it is for the LLM compared to a human, because we can't ascribe difficulty/ease to anything the model "does", simply because we know the fundamental mechanics of its inferencing. We can only after the fact of an LLM's output say something like: "Wow that would have been hard for me to output" or "I could have written something similar in like 5 minutes." But these claims can only ever be counter factuals like this, because in reality the output of an llm comes out at a constant rate no matter what you prompt it with.

If you try to say more, you'll end up falling in weird contradictions: it would take an llm a lot longer to output 10 million 'a's than a human, so it must be "harder" for the llm to do that than a human.