That’s only 3 orders of magnitude off from today’s largest models like PaLM (5x10^11 parameters), a gap that’s narrowed by 3 orders of magnitude just since 2019.
Thank you for this information. I did not know this. But my view (I may be wrong), is that AGI is too resource-intensive to be within the reach of normal computing of the ordinary user for at least 2 decades.
> is that AGI is too resource-intensive to be within the reach of normal computing of the ordinary user for at least 2 decades.
Hardware is still accelerating exponentially in density, albeit a bit slower. What you're not considering is that algorithmic improvements in machine learning are outpacing hardware improvements.
For instance, NVidia recently revealed how to switch from 32-bit floats to 16-bit floats with no perceptible loss in effectiveness, and they're working on 8-bit floats next. That's a full doubling in number of parameters in your model in only a single step. Other improvements are refinements to language models themselves to reduce overfitting and boost effectiveness with fewer parameters.
Arguably a machine learning model will achieve parity with human neuron density, in terms of number parameters, within the next decade. What that actually means is unclear.
You’re entitled to your own opinion, of course, but why do you hold this view?
And why is “the reach of normal commuting of the ordinary user” a relevant bar — Google search (as an example) requires computation beyond the reach of normal computing of the ordinary user yet has still had a tremendous impact.
This is very impressive, but since a biological brain is so much more complicated, who could really make a solid guess? Probably no one right now.
PaLM is not an attempt at AGI, a parameter is not equivalent to a neural connection, an activation function is not equivalent to a neuron (of which you have many different types), biological connection patterns are much richer, and biological stimuli are not like slideshows of a single type of data, so...
I made no claims contrary to anything in your post; your response — none of which I disagree with — makes me worry that you are coming in with a preexisting belief and just looking for reasons it must be true.
That said, there are plenty of multimodal networks (ie not slideshows), and we know very little about the relevance to intelligence of the “richness” of neural connections, activations, etc. — but it’s inarguable that we’ve made great strides in scale alone.