The idea that models can't be copyrighted isn't far fetched. The basic idea is that models are created by an automated process not by a person.
The courts have already upheld that AI generated output is not copyrightable for this exact reason.
So if you do not buy that it applies to models then you would have to explain the difference between the process which outputs bits into a model's layers (aka training) and the process which takes bits into the input layer and then dumps out the subsequent bits of the output layer (inference /generation).
Then explain why that distinction is different in regards to the applicability of copyright.
I'm not sure that even the "AI generated output is not copyrightable" stance will be maintained - as long as "AI generated output" becomes big business. Same way copyright was invented and Sonny-Bono-extended to the max as long as content became big business.
In the model's case, though, it's even easier why it could be copyrightable, as a "baked" model is still created by people fine-tuning it, setting parameters and hardcoded stuff, training it with this or that set and excluding other, and so on.
For example music composed and rendered as audio by generative algorithms (something which doesn't even need AI, just some rules and stohastic processes) has been created and copyrighted just fine for decades...
All the arguments for why photographs are copyrighted would seem to apply. The photographer isn't painting the image, but his artistic input is still vital to creating the image. Same with training these models: the training is just an algorithm on some data, but choosing the right hyperparameters and training data is an artistic expression of the author, making copyright apply
For the same reasons this monkey photo cannot be copyrighted it is highly likely that AI generated art is uncopyrightable and that would also mean that models are. The fact that humans set up the systems which produce the art/models with the intention of getting an end results generally like the one they get is simply not meaningful to the copyright dispute.
The courts have already upheld that AI generated output is not copyrightable for this exact reason.
So if you do not buy that it applies to models then you would have to explain the difference between the process which outputs bits into a model's layers (aka training) and the process which takes bits into the input layer and then dumps out the subsequent bits of the output layer (inference /generation).
Then explain why that distinction is different in regards to the applicability of copyright.