[wpietri]

Theory of mind is super important here! Not just that of experts, but that of consumers. An actual author is always thinking about the audience. About what they'll find novel or interesting. About the best ways to inform, entertain, or delight them. About how to make them feel.

Merely producing "normal-looking" text (great and accurate phrase!) is an unclosed feedback loop. The songwriter here has to stop himself from driving that loop forward because he's so used to working through multiple drafts as he makes things work for his internal simulation of his audience. And generally, once an artist has something that works internally, they'll then start running it by actual other people to get their reactions. Up to and including testing material on full audiences.

Because LLMs lack internal simulations of audience response, they'll always be limited to producing "normal-looking" work.