If your "world model" only models a small portion of the world, I think the more appropriate label is a time-series model. Once you truncate correlated data, the model you're left with isn't very worldly at all.
You don't need to load the entire world model in order to be effective at a task. LLM providers already do something similarly described with model routing.