[sn41]

As a theory person who usually explains O notation using concrete numbers, the degree of the neural network in our brain is approx 7000. Taking approx 86 billion ~ 100 billion, this itself is a graph with approx 6x10^(14) edges - does AGI proponents really hope to be able to do this? I am genuinely curious to know : is there some simplifying assumption which makes things faster?

[zone411]

This comparison is not perfect for various reasons. For example, the average firing rate of neurons is pretty low. Most attempts at comparison are done using FLOPS, e.g. https://www.openphilanthropy.org/research/how-much-computati.... You may prefer this comparison using Traversed Edges Per Second: https://aiimpacts.org/brain-performance-in-teps/.

[zamfi]

That’s only 3 orders of magnitude off from today’s largest models like PaLM (5x10^11 parameters), a gap that’s narrowed by 3 orders of magnitude just since 2019.

How far away do you think we are, exactly?

[sn41]

Thank you for this information. I did not know this. But my view (I may be wrong), is that AGI is too resource-intensive to be within the reach of normal computing of the ordinary user for at least 2 decades.

[naasking]

> is that AGI is too resource-intensive to be within the reach of normal computing of the ordinary user for at least 2 decades.

Hardware is still accelerating exponentially in density, albeit a bit slower. What you're not considering is that algorithmic improvements in machine learning are outpacing hardware improvements.

For instance, NVidia recently revealed how to switch from 32-bit floats to 16-bit floats with no perceptible loss in effectiveness, and they're working on 8-bit floats next. That's a full doubling in number of parameters in your model in only a single step. Other improvements are refinements to language models themselves to reduce overfitting and boost effectiveness with fewer parameters.

Arguably a machine learning model will achieve parity with human neuron density, in terms of number parameters, within the next decade. What that actually means is unclear.

[zamfi]

You’re entitled to your own opinion, of course, but why do you hold this view?

And why is “the reach of normal commuting of the ordinary user” a relevant bar — Google search (as an example) requires computation beyond the reach of normal computing of the ordinary user yet has still had a tremendous impact.

[d_tr]

This is very impressive, but since a biological brain is so much more complicated, who could really make a solid guess? Probably no one right now.

PaLM is not an attempt at AGI, a parameter is not equivalent to a neural connection, an activation function is not equivalent to a neuron (of which you have many different types), biological connection patterns are much richer, and biological stimuli are not like slideshows of a single type of data, so...

[zamfi]

I made no claims contrary to anything in your post; your response — none of which I disagree with — makes me worry that you are coming in with a preexisting belief and just looking for reasons it must be true.

That said, there are plenty of multimodal networks (ie not slideshows), and we know very little about the relevance to intelligence of the “richness” of neural connections, activations, etc. — but it’s inarguable that we’ve made great strides in scale alone.

[d_tr]

In your previous comment you seemed to suggest that we should not be very far. Maybe I misinterpreted you.

I do believe that AGI is possible and that it does not have to resemble a biological brain though.

[Jack000]

You can make the same argument for cats, dogs and other mammals, which do have embodied intelligence but not the skills we typically associate with general intelligence (language, deductive reasoning, math, etc). Raw neuron count is only loosely associated with intelligence, which is highly variable even between individual humans with nearly equivalent neurons.

Our brains are made of tiny little animals because that's just how life evolved on this planet. It's not a given that this is the best or even a good way to approach the problem.

[CuriouslyC]

Raw neuron count is only loosely associated with abstract problem solving intelligence. A cat's neural network is incredibly well optimized for things that cats care about.

Brains are very well optimized for computation/energy (while also being self replicating and self repairing), the tasks researchers care about just aren't the ones evolution cares about.