[GuB-42]

My theory is that LLMs here are put in a situation that matches its training dataset, which is mostly fiction since besides Hiroshima and Nagasaki, nukes have never been launched in anger, and I guess the most reliable sources are highly classified.

So, to a LLM, it is a game, because almost everything in its training data treats it as a game, and it reacts accordingly.

Same idea when we see LLMs acting like AI villains from sci-fi literature. That's because it has been trained with sci-fi literature, and as the auto-completer it is, it will recognize the situation as one of these stories and will continue it accordingly.

LLMs are storytellers, their reasoning is based on words, not on the physical world. Many of the stories they tell are useful, but one must not forget that they are stories, there is no intent behind them.