[spuz]

The example given in the paper of an extremely scrambled text is:

> oJn amRh wno het 2023 Meatsrs ermtnoTuna no duySan ta atgsuAu ntaaNloi Gflo bClu, gnclcinhi ish ifsrt nereg ecatkj nad ncedos raecer jroam

It's pretty hard to unscramble as a human. I'll leave you to read the paper if you want to see the original text.

[thaumasiotes]

> It's pretty hard to unscramble as a human.

Is it? With essentially no knowledge of golf and no lookups, in less than a minute of work, I get this:

Jon [last name beginning with R, Rahm?] won the 2023 Masters Tournament on Sunday at Augusta National Golf Club, clinching his first green jacket and second career major

I guess it's possible that I made a mistake in unscrambling, but I like my chances.

[bravetraveler]

Was thinking the same. If the words are maintained and only the letters that are jumbled... it gets easier the further you go.

Forgive the pun, but "The words literally start falling into place"

If we assume a sensible sentence, there are only so many combinations that make sense. The complexity of decoding (as a human) feels greatly overstated.

It's computationally-expensive spell check. Not to dismiss the study/tech, it's neat to see the machine apply context too.

You got it right, by the way, minus the name. I didn't copy that part to check

Edit: another way to look at this, a lot of information is encoded in those spaces

[BugsJustFindMe]

> > It's pretty hard to unscramble as a human.

> Is it? With essentially no knowledge of golf and no lookups, in less than a minute of work, I get

A MINUTE! FOR 23 WORDS! Yes, the fact that you measured in units of the nearest minute for something so short is the sign of it being hard. Compare how long it takes you to read the unscrambled version.

[checkyoursudo]

A language model being able to do something that rarely has to be done in the first place within milliseconds or less that would take a human a minute to do just does not seem that impressive. (I tried it without reading any of these comments below, and I got everything except "clinching" in about 20 seconds.)

I don't know. I am having a hard time overcoming the likelihood that scrambled and cipher-encoded words/solutions are part of the training corpus, thus fully explaining the phenomenon.

If someone can get it to decipher something like the zodiac killer's cipher, then I might be more impressed.

[BugsJustFindMe]

That's ok. You get to be impressed or not impressed by whatever impresses or doesn't impress you. Maybe nothing is impressive! I'm extremely fine with a statement like that.

I question whether your bounding the human time to a minute is valuable here though. If the jumbled content were multiple pages long instead of only 23 words, would it be somehow more impressive despite the process being exactly the same?

> I am having a hard time overcoming the likelihood that scrambled and cipher-encoded words/solutions are part of the training corpus, thus fully explaining the phenomenon.

Scrambled words are part of my training corpus too, but it still takes me a lot longer than the machine, and I don't even need to give the machine a hint about what's going on. I just say "tell me what this says" and a moment later it does.

[thaumasiotes]

I measured in units of minutes because the largest unit below that is the second, which is too small. How accurately do you think people can measure how long it takes them to do mental work?

I would usually determine whether to call something "hard" by reference to a measure of difficulty such as rate of success, not by whether doing it with no practice is slower than doing a similar task that I've practiced extensively.

[BugsJustFindMe]

> I would usually determine whether to call something "hard" by reference to a measure of difficulty

How long it takes you to do something compared to something else is a measure of its difficulty, all else being equal.

> such as rate of success, not by whether doing it with no practice is slower

Then I guess nothing is hard if you can ever eventually succeed, even if you struggle along the way, which sounds to me like not a very useful distinction.

Because that's what you're describing here. You're rapidly failing to interpret each scrambled word as its unscrambled form. You're sampling letters, failing, and trying again, over and over, until you eventually succeed, and then moving on to the next word. Maybe you're even backtracking to previous words that you got wrong (now/won perhaps) based on later unscramblings. And you're ignoring that part and only evaluating the very final outcome in a binary "got to the marathon finish line" fashion while ignoring the shortness of breath and stitch in your side.

The entire reason it takes longer is because you have a low rate of intermediate success, which makes progress slow, even though you got there in the end.

> I measured in units of minutes because the largest unit below that is the second

"Seconds" is an extremely common descriptor for how long something might take. But you didn't say "in seconds". This arbitrary rule about whole units sounds defensive. It's really ok for us to acknowledge the significance of the fact that reading the scrambled version takes significantly more mental effort.

[beepbooptheory]

Could you explain what your issue is here? I think we are generally just try to reason through this phenomena, not make grand conclusions about the model. We talk about how hard/time consuming it is for a human to pose possible theories for what the LLM could be doing. It is not to assert anything about how "difficult" it is for the LLM compared to a human, because we can't ascribe difficulty/ease to anything the model "does", simply because we know the fundamental mechanics of its inferencing. We can only after the fact of an LLM's output say something like: "Wow that would have been hard for me to output" or "I could have written something similar in like 5 minutes." But these claims can only ever be counter factuals like this, because in reality the output of an llm comes out at a constant rate no matter what you prompt it with.

If you try to say more, you'll end up falling in weird contradictions: it would take an llm a lot longer to output 10 million 'a's than a human, so it must be "harder" for the llm to do that than a human.

[bravetraveler]

Don't forget to compare reading and digesting to make it fair! Speed reading doesn't count /s

Apparently 'reading' (glancing at) something a few times is difficult work.

I can give you a bottle of water if you're tuckered out, I didn't need mine.

[freeqaz]

> Unscramble the following text: oJn amRh wno het 2023 Meatsrs ermtnoTuna no duySan ta atgsuAu ntaaNloi Gflo bClu, gnclcinhi ish ifsrt nereg ecatkj nad ncedos raecer jroam

The unscrambled text is: "Jon Rahm won the 2023 Masters Tournament on Sunday at Augusta National Golf Club, clinching his first green jacket and second career major."

[peterleiser]

I copied the scrambled text from the PDF but extra letters were inserted, spacing was messed up, etc. Typical PDF nonsense. So I took a screenshot of the letters, uploaded that to chatgpt-4, and it transcribed the letters. Then I asked it to unscramble them, which it did correctly.

[Gys]

I asked gpt 3.5 to scramble the original text, resulting in "wno hRaJ tnoh 3202 sretsaM tnemenoruT no yadnuS ta agustA latsignaN floN lbuoC ,bulcinnhc sih tsrif neerg tekcaj dna dnoces raecc romaj."

Interesting how it mixes across words.

[VMG]

or feed it to ChatGPT!

by the way, ChatGPT 4 is pretty good at scrambling as well. Your comment scrambles to

> tI's ryteppt darh ot cuaslmbern sa a amunh. l'lI eavel uyo ot dera eht erpap fi uoy nwat ot ese eht rliongia xett.

[xondono]

Kind of, but it did hallucinate an extra p for “pretty”