What's the difference between using a pencil to write something and using an LLM to write something? Seriously, I'm asking the question. Why does one produce something copyrighted why the other doesn't?
The copyright office has issued guidance on this which contains a very thorough and thoughtful legal analysis; you would probably be most interested section 3: https://copyright.gov/ai/ai_policy_guidance.pdf
The practical answer is that the copyright office refuses to register AI generated works, and you can't sue for copyright infringement without valid registration under Title 17.
> What's the difference between using a pencil to write something and using an LLM to write something?
The pencil is not a derivative work of a pile of copyrighted material.
> Why does one produce something copyrighted why the other doesn't?
There's existing case law that non-human entities (e.g. animals) can't create copyrightable works. And in the case of an AI LLM, the AI LLM itself is a derivative work of its training data (as evidenced by the fact that it can by default spit out training data verbatim, even if it has had after-the-fact filters added to prevent such responses).