|
|
|
|
|
|
by keskival
1132 days ago
|
|
|
Large Language Models aren't a silver bullet – they don't solve all your problems. But they are a holy grail – as a universal common sense module they give IT systems a capability they never had before, a capability which has been sought after from since computers became a thing, a capacity for common sense. We now have that capacity and that alone will revolutionize the world. The chatbots aren't about chat, they are about common sense. Like the article, I am only talking about technology that already exists although the progress in deep learning is still super-exponential. We will certainly achieve AGI during this year as it will only require making these systems self-play like we did with AlphaGo -> AlphaZero -> MuZero. Self-play, or reinforcement learning with machine feedback will skyrocket the performance of these systems in language domain, which conveniently encompasses much of what is still missing for AGI. |
|
|
LLMs act in insensible ways all the time. They contradict themselves. They hallucinate. If you ask an LLM to follow a simple but long logic puzzle and show you its work, it will often make extremely obvious errors and fail to notice even when you ask it to review its work.
What LLMs can do is coherently string together language. That requires sophisticated linguistic understanding, and LLMs are pretty impressive for it. But merely understanding language is not intelligence, or even common sense. I suspect we're approaching the limits of what we can get out of statistical language generation.
Actual AGI would require a logical model of the world, not a probabilistic model of language. Looking at non-human animals, we can see problem-solving evolved long before language did. Language is a second-order phenomenon we use to express first-order problem-solving conclusions about the world, and I'm skeptical that we'll ever manage to accurately recreate first-order problem-solving by training models on the second-order linguistic artefacts of problem-solving. It's actually very easy to create superficially-realistic second-order problem-solving artefacts which, when examined with first-order problem solving capabilities, don't stand up to scrutiny—i.e. contradictions, hallucinations, and faulty reasoning. I suspect that, when computers do learn to problem-solve, it will have been by training to solve problems.
There's also no way we'll get the kind of growth you're predicting from self-play. When it comes to a simple competition like a board game, it's easy to optimize for more skilled play. But there's no objective way to win a conversation. Maybe we'll get some gains out of training these systems against each other, but it's just not a clearly viable use case.