[pmayrgundter]

Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here.

Training one of these big models takes 100kWh for 1e19 flops, so that's 100k Wh, 360M Ws, or 360MJ or 3.6 1e8J. 1e8Joules/1e19flops = 1e-11J/flop

Neurons take 1e-8J/spike.[1]

Math check appreciated :)

Does seem plausible to think of a single neuron spike (hodgkin-huxley cable model) being modeled with ~1k flops. Though I'm firmly of the opinion that nobody really knows how the brain works.. the neural spike activity could be pure epiphenomenon.. who knows!

[1] “Finally, the energy supply to a neuron by ATP is 8.31 × 10−9 J. Meanwhile, integrating the total power with respect to time we will get the consumed electric power, which is 8.75 × 10−9 J. This is more energy than the ATP supplied. The energy efficiency is 105.3%. This is an anomaly…” - 2017 Feb 16 Wang, Xu, Institute for Cognitive Neurodynamics, East China University of Science and Technology https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5337805/

[whatshisface]

The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.

[pmayrgundter]

Sorry, didn't mean it quite like that. It's clear neural spike activity exists as a physical process. I'm suggesting that spiking activity may be an epiphenomena more primary brain functions, i.e. information processing, consciousness, etc..

As far as I know, we're closest to showing information processing in the visual cortex (which is highly linear) and we're still a long way from knowing how it works at a neural level. But maybe someone here can update on this?

But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems. But even if so, that's just "information processing".

I'm personally a fan of electromagnetic theories of consciousness[], where the synaptic activity could be an epiphenomenon of supporting a stand EM field.

[]https://en.wikipedia.org/wiki/Electromagnetic_theories_of_co...

[whatshisface]

>But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems.

I am not sure how much is known about information processing, but it's clear that motor impulses and sensory information are encoded in the spikes. Higher spike frequency = stronger signal. Synapses are how signals are passed from neuron to neuron.

[pmayrgundter]

Ok, that's fair. That's i/o and yes, that's known to be highly linear by the time it gets to the efferent nerves, and makes sense it is before that as well. I think that still leaves the vast majority of the cortex using undefined mechanisms.

[whatshisface]

There's no need to hypothesize a wholly unique central nervous system signalling mechanism when, not only is the signalling mechanism of peripheral nerves understood, central nerves are observed doing the same thing.

[pmayrgundter]

I think it's fine as a hypothesis for the CNS, and my guess is it's correct for the spinal cord on up to the maybe the thalamus. But there the anatomy changes radically, as does the electrical activity.. eg the cluster waves (alpha, beta, theta) begin there and indicate some sort of group behaviors in various areas.

Afaik, we have correlative descriptions of what these waves indicate (importantly, they're associated with sleep, consciousness, attentiveness), but no direct mechanical model of them or a clear purpose. So yeah, spike timing could still be used at this level, but it seems other behaviors are also happening that may be more essential to the larger function.

[posterboy]

I do remember reading once that glia cells are not understood, and the something about how electro magnetic fields might also induce ... I do not remember it well because it wasn't very specific

The sentiment that synapses probably don't explain everything is rather common, anyhow. I'm thinking, the way the blood flow literally influences the relevant areas by transporting available energy for example, and neurotransmitters must be a very important point, and how those areas react in case of insufficiency would explain why I become nasty when tired and hungry at the same time.

[dekhn]

For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics.

See more: https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...

[whatshisface]

I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.

[netizen-936824]

That isn't quite true

https://en.m.wikipedia.org/wiki/Consciousness

[dekhn]

No. https://en.wikipedia.org/wiki/Epiphenomenalism

[whatshisface]

Note that that title ends in "ism," like "Calvinism," or "Evangelicalism."

[posterboy]

hence epiphenomenal-ize? I don't thinkso.

Maybe it's more like epiphenomalis-m(a), where -is could be a genetive ending. I.e. the idea of epiphenomena.