[skybrian]

There are probably a lot of applications where the LLM could rely more on data that's supplied to it just-in-time in the context window, and less on specialist knowledge from its training set.

Also, "natural" data taken from the Internet is probably quite inefficient as training material. It's going to have a lot of duplication. You only need each fact once to be able to synthesize more examples of it.