[rklaehn]

But if the AI has been trained on open source licensed code and occasionally can produce verbatim copies of said code, shouldn't the weights of the AI then also become open source under the same license?

You could argue that training is a form of compilation, and the weights are a derivative work.