|
|
|
|
|
|
by semi-extrinsic
1982 days ago
|
|
|
> It is fairly obvious to many now, after the continued scaling of the GPT-x series models, that genuine intelligence is an emergent property of the kind of systems we are building. I respectfully disagree. GPT-x series models are performing interpolation on an unfathomably massive corpus. It is not hard to find cases where it directly reproduces entire paragraphs from existing text. When given a prompt on a topic for which it finds multiple existing texts with similar degree of matching, such as different articles reporting on the same topic, it is able to blend the content of those articles smoothly. I mean, GPT-3 is around 6 trillion bits of compressed data. The entire human brain has 0.1 trillion neurons, and it obviously has a capacity far beyond GPT-3 - even in the extreme case if we assume all the neurons in the human brain are used for generating English written text. In my view GPT-x is very, very far from any kind of general intelligence. |
|
|
Cool :)
> The entire human brain has 0.1 trillion neurons
You want to be thinking about synapses. There's about 7000 synapses per neuron, so that's 7000 * 0.1 = 700 Trillion synapses. So thats *100 times larger than GPT-3. Also consider that a neuron does a fair amount of processing within the neuron, there is some very recent research on this, each neuron is a akin to a mini neural network. So I would not be surprised if the human brain is 10,000 times more powerful than GPT-3.
> It is not hard to find cases where it directly reproduces entire paragraphs from existing text. When given a prompt on a topic for which it finds multiple existing texts with similar degree of matching, such as different articles reporting on the same topic, it is able to blend the content of those articles smoothly.
This may be true, but it does not prove your hypothesis that all GPT-x models are simply "performing interpolation". Also the ability to perform recall better than a human may be to do with the way that we perform global optimisation over the network, rather than the local decentralised way that the brain presumably works. Point is accurate memorisation does not preclude general intelligence. Spend some time with the models, sit down for a few hours and investigate what they know and do not know, really look, see beyond what you expect to see. You may be surprised.