[jessriedel]

For those who are curious, a typical American home uses of order a kilowatt, time-averaged (10,400 kWh per year = 1.2 kW). So 30 MW is roughly the average power usage of a city of 30,000 homes, or 80,000 people, although total capacity will be larger to handle fluctuations.

[cr0sh]

...and yet, even a machine at this scale, or even 100 times that, could not come close to being on par, in terms of neural/neuron simulation, with that of the human brain.

We are definitely "doing something wrong" when it comes to artificial neural networking; even though are models are much simpler, it still takes an enormous amount of computing power, both in terms of raw CPU as well as actual electrical needs, just to be able to simulate things at a small scale (and if we use more accurate models, based on what we know about the brain and neurons, then at best our simulations can only be run to simulate, over our actual-time, what would be in actually fractions of a second in real-time).

That our brains can do so much using so little power (wattage), with such a high number of nodes and interconnections that dwarf anything we've so far have managed to simulate - it's a bit mind-boggling and humbling.

I just wonder where and what the issue actually is.

Why do our current practical models of a neuron, which are vastly simplified, require so much power to run at scale?

Is the issue related to the fact that they are simplified models, and actual neurons with their complexity are able to do things we don't yet understand or know about?

All of this is also related to back-propagation; such a thing doesn't seem to exist in nature (jury is still out on the theory, though) - so how do biological neural nets "learn"?

If we could eliminate or reduce the need for backpropagation, would that lower our power requirements for artificial network implementations?

As someone who has merely dabbled with artificial neural networks, these questions and conundrums fascinate me, and cause me to attempt to think up potential solutions, however far-fetched.

I highly doubt I will be the one to solve the issue, but I do hope to see it solved within my lifetime.

[uj8efdjkfdshf]

The main difference between the brain and a CPU is that the brain runs at a much lower, variable clock speed (10-100Hz) to reduce switching costs and makes up for this with extensive pipelining and parallelization when possible. The high node count and necessary connectivity is then possible due to the use of directed self assembly.

At any rate, it is quite likely that the neocortex of the brain simply computes a function(s) recursively upon sensory input (see Chomsky's minimalist program for suggestions on what it could be). What is unclear is what this function is, and how it comes about - for this reason the approach by some has been to attempt to simulate an entire brain to see what it does. But without the necessary abstractions, this will be inherently wasteful and generates nothing new other than validating your experimental data.

[kevin_thibedeau]

Connectivity. ICs are more or less 2D structures. That leads to less efficient connectivity and packing.

[srcmap]

Probably current AI is base on FP64, FP32, biological neuron likely be large network of only one or few bits. Once we figure out how to build, use, optimized large network of nodes which only process a few bits each, the power consumption might just go way down.

Someone might also find that kind of network can run a KHz like human brain cells instead of MHz, GHz. The power usage will go down even more.

[p1esk]

It seems you’re confusing AI and brain simulation.

[dekhn]

brains can't run high resolution simulations of brains, but supercomputers can.

neural networks are only a small component of what we use supercomputers for.