[hydrolox]

I think that shows how LLMs lack an important part of what we as intelligent agents have, as the parent comment pointed out, the innate ability to have some sort of train of thought or self check mechanism. In a human, as you said, we don't immediately blurt out or write out a phrase, since if someone gave us this problem we would immediately start considering the constraints and possibilities. By contrast, LLMs do not have this and the ability has to be "bolted on" through a pre prompt like "consider if your answer is correct" and "think step by step" etc. As far as actually choosing a viable strategy, such as figuring out to go alphabetically, this probably emerges as they get larger and larger; i.e, they need both the bolted on train of thought ability and also an actual good sense of reasoning and logic, which could be compared to a person (maybe a child) who can't come up with a good solution, let's say, to making a pangram (in this case).