[ftxbro]

For the lower bound there is a dark horse factor that has spooked Geoffrey Hinton. He thinks that biological brains aren't able to do backpropagation effectively through multiple layers, and so differentiable programming frameworks are much more powerful than what the brain has, at an algorithmic level. In other words, he thinks that computers are able to learn more effectively than any neuron-based biological brain. Of course right now there are caveats. The brain appears to have more 'statistical efficiency' meaning it appears to learn more from less data, and the brain is obviously more energy-efficient. There is also the possibility that Geoffrey Hinton is just wrong.

[moffkalast]

Biological brains also don't really operate layer by layer and can have connections between random neurons, so it's probably a lot more space efficient. Impossible to say if any of that actually matters though.

[DoctorOetker]

Can you cite a reference for Geoffrey Hinton (and or others) on this line of thought on this "dark horse factor"? I think this resonates with a lot of people, having a focal point Schelling point to refer to in discussions would be handy.

I believe the situation is a lot more extreme than the absurd efficiency of differentiable programming. I have been meaning to write up (but been too busy to do so) an insight where I believe training can be made ridiculously cheap computationally speaking (in a way that combines with differentiable programming, not replaces it). I am agnostic if this is what the brain does, but wouldn't be surprised at all if the brain does in fact do back-propagation (or uses the insight that I've been meaning to write up).