I suspect that part of the OP's point was that ChatGPT happily parrots the aggregate critical opinion as its "thoughts" despite never having heard the song (or parsed its MP3 file or stems or sheet music)
If you ask me what I think of a song I've never heard, I'm general enough to want to listen to it...
So? Do you think we can’t make an LLM which picks a favourite song and writes about it with the gusto a person does? What is this example supposed to illustrate?
We clearly can, but not with gusto. LLM can't feel gusto about anything. If the point is that it doesn't matter as long as a person reading it is convinced there is gusto, then my point is that the 'opinion' of such a thing is irrelevant.
This is circular reasoning. LLMs can't feel gusto because they can't feel gusto. You have any way to measure "gusto" that we're all aware of ?
If other humans ascribing the quality of something you can't properly define isn't enough then you clearly don't care about what a LLM does or does not have, only what you are convinced it doesn't have.
Maybe, but 1. the point isn’t to describe, but to explain an abstract and potentially novel idea (thought) on the song, and 2. we can’t train this kind of thing in a generic way right now.
If you ask me what I think of a song I've never heard, I'm general enough to want to listen to it...