[antli35]

If you want to do a good job of generating text, you have to develop a model of how the world works. For example, if you describe an experiment from a paper to ChatGPT and ask it to generate the results section of the paper, then ChatGPT probably needs to understand the phenomenon that the experiment is about and be able to model it to some degree in order to generate plausible results. If you think about ChatGPT in this way, then it is not just a text generator, but a world simulator. The more accurately you can simulate the world, the better you can generate text. I think this is where the model's size and complexity comes from. ChatGPT needs to know as much as it can about basically everything.

Putting it more generally, the difficulty of a computation isn't necessarily correlated to the filesize of the end product of that computation. Imagine simulating the entire world to try to predict what next week's lottery drawing numbers are going to be. Would require an unimaginable amount of data and computation, yet the output will be just a couple numbers.