[rad88]

It doesn't seem that crazy. Working from a simpler analogy first -- if an alien can read 95% of written English, but completely mischaracterizes the remaining 5%, would a reasonable scientist conclude that they can't read at all and are just guessing randomly? No, and it would indeed be unreasonable to conclude that there's no way to write differently so that the alien performs substantially better.

We know some more about LLMs than we do aliens. The LLM is a neural network whose parameters have been optimized to reduce the total size of its errors, as measured against an enormous set of empirical data.

We have to add to your analogy then, that it's somehow known the alien speaks English on their own planet remarkably well. They are still not perfectly correct, but when they have to describe something on that planet, in English, they can do it better than they generally can given the same task on Earth.

It would again be totally unscientific to conclude that because of this, there is no way the alien can talk about earthly objects or ideas. The impulse of the mere engineer, to just hack around and find out, is a scientific one.