[TaylorAlexander]

sigh

Neural networks are not based on how neurons work. They do not copy aspects of us. They call them neural networks because they are sort of conceptually like networks of neurons in the brain but they’re so different as to make false the statement that they are based on neurons.

[Terr_]

*brandishes crutches*

"Behold! The Mechanical Leg! The first technology that actually copies aspects of our very selves! Think of what wonders of self-discovery it shall reveal!" :p

P.S.: "My god, it is stronger in compression rather than shear-stresses, how eerily similar to real legs! We're on to something here!"

[famouswaffles]

They are though. They quite literally are. Saying otherwise is like saying planes weren't based on how birds work when Wright brothers spent a lot of time in the 1800s studying birds.

Both Humans and GPT are neural networks. Who cares that GPT doesn't have feathers or flap its wings? That's not the question to care bout. We are interested in whether GPT flies. You can sigh to Kingdom come and nothing will change that.

We've developed numerous different learning algorithms that are biologically plausible, but they all kinda work like backpropagation but worse, so we stuck with backpropagation. We've made more complicated neurons that better resemble biological neurons, but it is faster and works better if you just add extra simple neurons, so we do that instead. Spiking neural networks have connection patterns more similar to what you see in the brain, but they learn slower and are tougher to work with than regular layered neural networks, so we use layered neural networks instead.

The Wright brothers probably experimented with gluing feathers onto their gliders, but eventually decided it wasn’t worth the effort. Because that's not what is important.

[ImHereToVote]

There are drones with feathers now however. The spring in feather flaps help conserve energy, but only in flapping wings obviously.

[robwwilliams]

If you study retinal synaptic circuitry you will not sigh so heavily and you will in fact see striking homologies with hardware neural networks, including feedback between layers and discretized (action potential) outputs via the optic nerve.

I recommend reading Synaptic Organization of the Brain or getting into if you are brave, the primary literature on retinal processing of visual input.

[TaylorAlexander]

Actually it’s funny my best friend is a neuroscientist and studies the retina and in particular the way different types of retinal cells respond to stimulus. I have watched her give presentations on her work and I do see that there are some similarities.

But it is nonetheless the case that “neural networks” are not called that because they are based on the way neurons work.

[ImHereToVote]

They literally are:

https://www.ibm.com/topics/neural-networks#:~:text=Their%20n...

[smokel]

The book "The Synaptic Organization of the Brain" appears to be from 2003. Is it still relevant, or is there perhaps a more recent book worth checking out?

[robwwilliams]

It is great even though older. The chapter by Sterling on retina is amazing. Yes there is an updated version by Gordon Shepherd and colleagues: Handbook of Brain Microcircuits, but I actually prefer the 2003 edition.

[__loam]

I will continue to sigh. The visual cortex is relatively simple and linear. You're not saying something that's as impressive as you think it is.

[l33t7332273]

I think the point of the example is that that is an important part of our brains that is relatively simple and linear and we’ve been able to mimic it.

[robwwilliams]

Anything but simple and anything but linear.

[martindbp]

Sigh... Everyone knows artificial neurons are not like biological neurons. The network is the important part, which really is analogous to the brain, while what came before (SVMs and random forests) are nothing like it.

[mecsred]

Sigh... Every man knows the mechanisms of the mind are yet unlike the cogs and pinions of clockwork. It remains the machinery, the relation of spring and escapement, that is most relevant. Hitherto in human history, I think, such structure has not been described.

[TeMPOraL]

If you build a neural network out of cogs and pinions, sure.

Comparing the brain to most complex machines in history wasn't a mistake, any more than refining laws of physics were. Successive approximations.

And we're no longer at the point where we're just comparing brain to most complex machines. We have information theory now. We figured out computation, in form independent of physical medium used. So we're really trying to determine the computational model behind the brain, and one of the ways to do it is to implement some computational models in whatever is most convenient (usually software running on silicon), and see if it's similar. Slowly but surely, we're mapping and matching computational aspects of the brain. LLMs are just one recent case where we got a spectacularly good match.

[TaylorAlexander]

> Everyone knows artificial neurons are not like biological neurons.

Not, apparently, the person I was replying to!

[ImHereToVote]

I'm him, and I didn't say that. ANNs didn't arise in a vacuum and they aren't called neural networks for the fun of it.

https://www.ibm.com/topics/neural-networks#:~:text=Their%20n....

[renewiltord]

Doesn't really matter to modern CS, but Rosenblatt's original perceptron paper is a good read on this. ANNs were specifically inspired by Natural NNs and there were many attempts to build ANNs using models of how the human brain works, specifically down to the neuron.

[dekhn]

I;m sure you know but one of the best ways to get neuro folks worked up is to say anything about neural networks being anything like neurons in brains.

(IMHO, Rosenblatt is an underappreciated genius; he had a working shallow computer vision hardware computer long before people even appreciated what an accomplishment that was. The hardware was fascinating- literally self-turning potentiometer knobs to update weights.

[renewiltord]

If I'm being honest, I do know they get annoyed by that stuff but I've never really understood why. It's a somewhat common pattern in Mathematics as an avenue for hypotheses to take an existing phenomenon, model some subset of its capabilities, use that to define a new class of behaviour, follow that through to conclusions, then use that to go back to seeing if those conclusions apply to the original phenomenon.

A theoretical such thing might be for us to look at, say, human arms and say "Well, this gripping thing is a cool piece of functionality. Let's build an artificial device that does this. But we don't have muscle contraction tech, so we'll put actuators in the gripping portion. All right, we've built an arm. It seems like if we place it in this position it minimizes mechanical wear when not in action and makes it unlikely for initial movement to create undesired results. I wonder if human arms+hands have the same behaviour. Ah, looks like not, but that would have been interesting if it were the case"

Essentially that's just the process of extracting substructure and then seeing if there is a homomorphism (smooshy type abuse here) between two structures as a way to detect yet hidden structure. Category theory is almost all this. I suppose the reason they find it annoying is that there are many mappings that are non-homomorphic and so these are the false cognates of concepts.

Still, I think the whole "An ANN is not a brain" thing is overdone. Of course not. A mechanical arm is not an arm, but they both have response curves, and one can consider a SLAM approach for the former and compare with the proprioceptive view of the latter. It just needs some squinting.

Anyway, considering your familiarity with R and his work, I think I'm not speaking to the uninitiated, but I thought it worth writing anyway.

[lupire]

It's an ego thing and a focus thing. They put so much effort into studying the biological details, that they resent people exteacting value without going to that effort, and feel offended at the implication that what they are studying is simple.

Ultimately it's largely down to misapprehension of the difference between emulating a neuton and simulating a neuron, and defensiveness about an approximate model.

[robwwilliams]

Hmm, in my experience this is not the primary reason, and I would also argue that the original statement is wrong. Computational neuroscientist are very comfortable (too comfortable perhaps) with simplified models of neurons and brain circuits. We accept them as stick figures. DNNs fit into the typical mindset of synaptic and neural circuits. Reading Synaptic Organization of the Brain should leave you stunned at the relative simplicity of how neuroscientists think the brain works.

Most neuroscientists do not indulge in deep thought because it does not pay the bills. NIH does not push us toward “understanding cognition and consciousness”. They push us toward treatments for Alzheimer’s, schizophrenia, stroke, epilepsy, and addiction. We do not have time for the fun stuff until we are ready to retire, which is when many of us become terrible neurophilosophers, e.g., Eccles.

[ImHereToVote]

Science history should be mandatory for undergrads. I didn't think what I said is controversial. This is established history. Sorry if it scares you.