Why is that a prerequisite? There are entire philosophies about what makes good design for UI's etc. Why can't models figure this out? Why do you feel this is some sort of mystical thing out of reach?
If you think all of the complexity of the human experience is reducible to statistical weights between tokens, that’s fine. Go with God. I don’t think that it is.
What do you think humans are? What’s the mechanism by which we make decisions, learn, remember, etc? Why would this be anything more than a very complex system that has been slowly optimized via evolution to perform well in its environment over long time horizons?