[gwern]

> You are, however, missing the point.

No, you are. You are making a lot of hay out of a simple, dumb, trivial technical flaw, which is easily fixed, and reading deep lessons and 'sadness' into it. It's not interesting at all (which is part of why I'm angry I have to waste so much of my life explaining to people why their character-based gotcha is actually uninformative and irrelevant).

It no more tells us anything about deep learning or human intelligence than would waving a piece of paper 'ABC is to XYZ as etc' in front of a blind person and them promptly failing the test, even if they are a skilled song writer. Duh! Of course they fail the test! They're blind. Their eyes don't let them see the individual characters, anymore than a BPE tokenization lets a LLM see individual characters. You would be a fool to conclude that 'blind humans are not actually intelligent, and yet, he wrote such good lyrics, isn't it sad that the parts of T.S. Eliot we cared about were just some sort of wiggling dog leg' on the basis of failure due to such an obvious mechanical perceptual flaw. Maybe blind people are less intelligent on average due to their handicap, sure, maybe human intelligence is in fact a sad dog leg wiggling around - both of those are entirely possible - but you need a different measurement and it should have been obvious to you and everyone else before you wasted your time waving the paper. The waving paper told you nothing of interest whatsoever.