While true, there’s a relatively small upper bound on how many bits of information are in this pre-training. Specifically, in the form of how much information is contained in DNA, which is only a couple gigabytes.
Stable diffusion model is around 4 gigabytes, inside that 4 gigabytes you have understanding of the whole english language model and mapping to billions of objects, people, concepts etc capable of generating from just a single sentence almost any picture in any style you imagine. Seems like a few gigabytes can hold a lot of information.