If a person plays a song similarly enough, then it is copyright infringment! Mere knowledge is irrelevant, it is the producing of copies (and also a few related actions) which is prohibited by copyright.
No language model plays a song either in the narrow sense, they just send a representation of the song to some other program (or human) that might play it.
Mere knowledge is irrelevant only because we don't (yet) have a mechanism to pry open one's brains and inspect the copying of songs within different parts of one's brain. Otherwise, mechanistically, besides one using silicon and other using wetware, they're pretty much doing the same thing.
That's my point. If you could pry open a human brain and decipher how it works, you'll see some representation of the song being sent around to various parts of the brain.
Mere knowledge is irrelevant only because we don't (yet) have a mechanism to pry open one's brains and inspect the copying of songs within different parts of one's brain. Otherwise, mechanistically, besides one using silicon and other using wetware, they're pretty much doing the same thing.