It’s still exponential, but a little slower. (edit: wait, is that still exponential if it slows down?) Anyway we only need to get to human level (or maybe a bit less) and we’re not that far off (maybe 10 or 20 years at current rates of progress?)
Not all types of AI need external training data, you can train on how effectively a goal is achieved
> they've parasited, sorry, trained on the entirety of accessible human knowledge
I see this as a new development in language, used to be restricted to meat neural nets and books, now it can also be consumed and created by LLMs. A new self replication path was opened for language. Language is an evolutionary system, it's alive. Without Language humans are mere shadows of what they can be. Language turns a baby into a modern adult, and a randomly initialised neural net into chatGPT.
The magic was always in the language, not in the neural network. We should care more about the size and quality of the training dataset than the model. Any model would do, all model tweaks are more or less the same. But the data, that is the origin of all the abilities. But we cannot own abilities, it should be fair game to learn abilities and facts even from copyrighted data. Novel and creative training examples should not be reproduced by LLMs, but mere facts and skills should be general enough not to be owned by anyone.
By your logic, just pick any random bum off the street, give him the right training set, then he will become a 180 IQ genius and discover the unified theory of gravity and quantum mechanics.
Some models are just inherently better at modelling.
The training data thing is a problem mainly for LLMs, so it might be a limitation if we purely scale up LLMs but there are other types of AI around too
Chip scaling still seems to be going pretty fast, and we may discover new ways to make better use of the chips we currently have, like better methods of quantisation, or just using more of them, which could get us just far enough to reach the self improvement threshold
So we could end up hitting a wall with chip scaling or something but I don’t think it’s that likely
Really? Even a 5% generation-to-generation improvement would be exponential, it’s just 1.05 to the power of the generation. If it was linear you’d have benchmark results scaling by a fixed number of points each generation, which doesn’t seem to be a thing as far as I know
Not all types of AI need external training data, you can train on how effectively a goal is achieved