[StevenWaterman]

I thought about this more and realised your question might have been "what's the difference between knowing and learning". IE, how can we say the model believes something without having been taught it.

I think you're right that they're basically the same thing. I'd argue they're very slightly different because what an AI model ends up knowing isn't perfectly predictable based on what they were taught (emergent intelligence), but the sentence you quoted is using believing and learning to mean the same thing, it's just trying to draw attention to the fact that the training process structurally enforces "cheat as much as possible without getting caught".

IE, the contrast in the original sentence wasn't "believe" vs "learn", it was "good" vs "permissible"