Language models exist not in the physical world but digital. A more apt comparison would be if LLMs can compile new (or slightly modified) LLMs, and I believe the answer will soon be (if not already), a resounding yes.
I agree, LLMs can generate the source code for their network, can generate the training data, supervise a model being trained and explain step by step what they do. That's why I specifically said they can't make GPUs, if they could, they would be self replicators. We'd have to consider them alive.