| > As Hinton says, there is no real limit to how sophisticated they can get. There’s no limit to how sophisticated a model can get, but, 1. That’s a property shared with many architectures, and not really that interesting, 2. There are limits to the specific ways that we train models, 3. We care about the relative improvement that these models deliver, for a given investment of time and money. From a mathematical perspective, you can just kind of keep multiplying the size of your model, and you can prove that it can represent arbitrary complicated structures (like, internal mental models of the world). That doesn’t mean that your training methods will produce those complicated structures. With Go, I can see how the model itself can be used to generate new, useful training data. How such a technique could be applied to LLMs is less clear, and its benefits are more dubious. |