[pessimizer]

Really? I'd assume that an LLM would deduplicate Wikipedia into something much smaller than 25GB. That's its only job.

[crazygringo]

> That's its only job.

The vast, vast majority of LLM knowledge is not found in Wikipedia. It is definitely not its only job.

[Tostino]

When trained on next word prediction with the standard loss function, by definition it is it's only job.