[EncomLab]

Yes that is the definition - but that is not what is occurring her. We DO know exactly what is going on inside the system and can determine precisely from step to step the state of the entire system and the next state of the system. The author is making a claim based on woo that somehow this software operates differently than any other software at a fundamental level and that is not the case.

[HarHarVeryFunny]

Are they ? The article only mentions "black box" a couple of times, and seems to be using it in the sense of "we don't need to be concerned about what's inside".

In any case, while we know there's a transformer in the box, the operational behavior of a trained transformer is still somewhat opaque. We know the data flow of course, and how to calculate next state given current state, but what is going on semantically - the field of mechanistic interpretability - is still a work in progress.