I'm not even in favor of banning/heavily-regulating AI developments, but I think this position here is a little reductive; you could boil anything down to the point of absurdity.
The point of nuclear weapons bans, for instance, isn't to control "evil atoms from touching", it's to prevent the higher-order effects that those atoms touching can cause.
Very funny, the problem is more about the input than whatever the CPU is doing, as in chatGPT would be no more than a footnote without copyrighted material in all it's datasets.
They wouldn't have used copyrighted material if it wasn't a survival matter for them, but sure you go ahead and pretend that they just happen to use it and not really need it that much.
Anyone that doesn't at least acknowledge copyrighted materials will be a footnote. But you don't need it for AI's most profitable applications at all, eg. code generation or sentiment analysis.
The point of nuclear weapons bans, for instance, isn't to control "evil atoms from touching", it's to prevent the higher-order effects that those atoms touching can cause.