[jakobson14]

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers")

Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong."

Neurologists seem to have a really hard time comprehending that researchers working on neural networks aren't as clueless about computers as neurology is about the brain. They also vastly overestimate how much engineers working on neural networks even care about how biological brains work.

Virtually every attempt at making neural networks mimic biological neurons has been a miserable failure. Neural networks, despite their name, don't work anything like biological neurons and their development is guided by a combination of

A) practical experimentation and refinement, and

B) real, actual understanding about how they work.

The concept of resnets didn't come from biology. It came from observations about the flow of gradients between nodes in the computational graph. The concept of CNNs didn't come from biology, it came from old knowledge of convolutional filters. The current form and function of neural networks is grounded in repeated practical experimentation, not an attempt to mimic the slabs of meat that we place on pedestals. Neural networks are deep because it turns out hierarchical feature detectors work really well, and it doesn't really matter if the brain doesn't do things that way.

And then you have the nitwits searching the brain for transformer networks. Might as well look for mercury delay line memory while you're at it. Quantum entanglement too.

[robbrown451]

I can't agree with the dismissiveness of this comment, and frankly I find its tone out of line and not with the spirit of Hacker News.

There are insights that can come from studying the brain, that do indeed apply. Some researchers may not glean anything from such studies, and some may. I have no doubt that as neural networks get more an more powerful, we will continue to find more ways they are similar to the brain, and apply things we've learned about the brain to them.

I certainly prefer to see people making comparisons of neural networks to the brain, that the old "it's just a glorified autocomplete" and the like.

Relax.

[ramraj07]

No one disagrees we might be able to discern insights if we understand how our brain is wired. The problem is the current state of neuroscience is so flawed in its approach it’s not looking like they’re of any use. They don’t even understand how a 900 neuron worms system works but are more than happy to tap half a billion dollars from unsuspecting politicians saying they’ll map the human connectome. Go read the brain initiative proposal [1] to see how out of touch with reality the scientists in this field are. I agree with OP that sharp criticism of the entire field is fully warranted.

1. https://braininitiative.nih.gov/sites/default/files/document...

[andbberger]

what are you talking about is this konrad kording's shitposting alt??? this reeks of naivety

I certainly have many critiques of methods used in neuroscience rn (as a working neuroscientist) but to reduce those to the conclusion that the entire project of neuroscience is hopeless is absurd. We understand certain things quite well actually, and it's not at all obvious what "understanding" at a larger scale would look like. It is very possible that the brain is irreducibly complex, and that the model you would need to construct to describe it would itself be so complex as to be useless in providing insight. Considering that the brain is by far the most complex object in the universe I think we're doing pretty well.

Furthermore, there are quite a lot of disagreements about the utility of connectomics. Outside of the extremists (Sebastian Seung and his ilk) no one thinks that connectomics is going to be the key that brings earth shattering insight. It's just another tool. There is a complete connectome for part of the drosophila brain already (privately funded btw), which is in daily use in many fly labs. It tells you what other neurons are connected to. Incredibly useful. Not earth shattering.

also you might want to measure the neuroscience funding you deem wasteful up against the tens of billions NASA is spending to send humans (and not robots) back to the moon for "the spirit of adventure". cold war's over. robots will do just fine for the moon.

[ramraj07]

Can you please elaborate what great strides the field of neuroscience has made in the past 30 years?

From where I stand I can’t see anyone giving a clear explanation of anything our brain does or does not do in a disease. The only novel treatment that has come out seems to have been stick a rod into the brain and zap it and it just magically cures a lot of diseases we still don’t understand even a bit.

This is not even starting to discuss what little we have learned about how brains algorithms work. I’m still waiting to understand why pyramidal neurons were somehow groundbreaking. We found some neuron that fires when you walk to a place, why wouldn’t we find one?

And what are you saying about the fly connectome again? Do we have exact names for every neuron in the fly brain and its verified connectome for every neuron?

Last I checked the worm connectome has been available in intricate detail for decades and the scientists still haven’t had any proper decoding of the algorithms in that system. In fact I know every lab trying to figure that out now, I wrote proposals in the topic myself. Everyone else has apparently decided it’s not sexy enough to work with worms so they have just leaped to more complex systems with no basic understanding. I’m not the only one saying this. Sydney Brenner said as much in an editorial. But the field was too busy doing I don’t know what to listen.

Sydney, B. & Sejnowski, T. J. Understanding the human brain. Science 334, 567 (2011).

I remember sauntering to the occasional neuroscience talk during my ut southwestern PhD and occasionally hearing some professor brag about how the majority of one of their PhD’s jobs was to segmenting a single neuron in the thousand EM images or something. Surely that’s a sign this field needs revision?

[andbberger]

> And what are you saying about the fly connectome again? Do we have exact names for every neuron in the fly brain and its verified connectome for every neuron?

onus isn't on me to justify the existence of an entire field to you. the claim that neuroscience has not made great strides in the last 30 years is an extraordinary one, and that's all on you. but it especially doesn't help your case that if you had googled "fly connectome " you would have seen that the first result is a complete connectome of a larvae and the third result is the tour de force from Janelia that produced an adult connectome. With names and verified connections. there is even a wikipedia article for the drosophila connectome!

> I remember sauntering to the occasional neuroscience talk during my ut southwestern PhD and occasionally hearing some professor brag about how the majority of one of their PhD’s jobs was to segmenting a single neuron in the thousand EM images or something. Surely that’s a sign this field needs revision?

and if you had gone on to actually read the hemibrain connectome paper you would have gained some appreciation for the gargantuan achievement that it was. it took hundreds of person years to generate ground truth segmenting neurons by hand, to develop the ML techniques required to automatically segment the rest (extremely difficult problem) and to then validate the automatic segmentations. not to mention the insane effort it was to acquire a half petabyte EM image of a single fly at sub-synaptic resolution in the first place.

I gotta hand it to you though, the position of naivety you've delivered your middlebrow dismissal from is truly impressive in magnitude.

[civilitty]

Agreed. Reading the GP’s comment it feels like it’s from bizzaro world. It’s the computer scientists who have been claiming that neural networks resemble the human brain - they even fucking named them neural networks for christ’s sake! That could be excused as naive hubris in the 1980s, it’s utter delusion now.

A surface review of neuroplasticity literature alone should free anyone of the illusion that “neural networks” have even a passing resemblance to biological neurons, something covered in neuroscience 101 and is widely internalized by its practitioners. The BS grant writing and PR scientists have to participate in is hardly reflect of state of the art science itself.

The irony is that machine learning methods are a perfect fit for neuroscience and biology in general which generates reams of data that is largely so multidimensional that manual analysis is intractable. What we’re seeing now is the crest of the academic hype cycle which - if the history of bioinformatics is anything to go by - means that ML will take years if not decades for the field to understand and filly utilize.

[bjourne]

Actually it was neuroscientists that developed the models nowadays used for machine learning. The McCulloch-Pitts neuron model introduced in 1943 which lead to Frank Rosenblatt's perceptron introduced in 1958. Machine learning algorithms mostly still use those models but computational neuroscience has progressed towards much more complicated neuronal models.

[anonymousDan]

It's typical of the arrogant, borderline anti-scientific attitude of a non-negligible fraction of the HN hive mind, i.e. if it came out of academia it must be a waste of time.

[SubiculumCode]

As another working neuroscientist, thank you. And cheers.

[__loam]

No I think these comments are quite necessary. People need to stop making these comparisons because they have absolutely no grounding in how brains actually work. There are bad ideas that should be dismissed.

[fastball]

Neural networks are absolutely based on a very simplified model of how brains work. Specific NN architectures are in turn based on specific parts of the brain (e.g. Convolution Neural Networks are based on the visual cortices of cats/frogs).

[andbberger]

nah, they're arbitrary function approximators that caught a lucky break. CNNs rose to prominence because natural scene statistics are translation invariant and convolutions can be efficiently computed on GPUs. and now that we have whole warehouses of GPUs, the current mood in DL is to stop building the symmetries of your dataset into the model (which is insane btw) and use brute force.

the tenuous connection DL once had to neuroscience (perceptrons) is a distant memory

[fastball]

A fabricated re-telling of the past, given that we didn't start using GPUs for this type of compute until the turn of the millenium.

[__loam]

If you want to talk about history, these things were invented using a 1950's understanding of neuroscience then promptly discarded until the ml people figured out how to make them useful.

[andbberger]

AlexNet was the turning point for DL.

[robbrown451]

You're saying the study has no grounding in how brains work? I'd think a more reasonable conclusion would be that the neuroscientists involved have no grounding in how artificial neural networks work.

It seems the whole point is to bring in additional details of how brains work, that the think may be relevant to artificial NNs.

[p1esk]

Artificial neural networks are the closest working model of a brain we have today.

Lots of graph nodes, with weighted connections, performing distributed computation (mainly hierarchical pattern matching), learning from data by gradually updating weights, using selective attention (and/or recurrence, and/or convolutional filters).

Which of the above is not happening in our brains? Which of the above is not biologically inspired?

In fact this description equally applies to both a brain and GPT4.

[aeternum]

Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model. Not to mention single-celled organisms that exhibit ability to navigate.

The model can be the closest working model but that doesn't mean it is complete. It's very likely that cells can store memories/information independent from weights.

[p1esk]

We can’t do that not because our mathematical neurons are too simple. We can’t do that because we don’t know the algorithms those biological neurons are running.

Do you see the difference?

[__loam]

There is of course a difference between the two things you say. They're both the reason we can't recreate the brain in software though.

[ben_w]

> Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model.

That's rather a bold claim given that artificial neural networks are universal function approximators.

[agalunar]

Impossible given that number of neurons.

It's perhaps not terribly surprising that it becomes possible with unlimited width or depth (or an arbitrarily complex activation function).

https://en.wikipedia.org/wiki/Universal_approximation_theore...

[simiones]

It's incredible to me how widely this is misunderstood.

The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function.

Additionally, the theorem only proves that for any given continuous function, there exists a particular NN with particular weight that can approximate that function to a given precision. Training is not necessarily possible, and the same NN isn't guaranteed to approximate any other function to some desired precision.

It seems pretty obvious to me that most interesting behaviors in the real world can't be modelled by a mathematical function at all (that is, for each input having a single output); if we further restrict to continuous functions, or step functions, or whatever restriction we get from our chosen activation function.

[simiones]

> Lots of graph nodes

Neurons are not connected by a simple graph, there are plenty of neurons which affect all the neurons physically close to them. There are also many components in the body which demonstrably affect brain activity but are not neurons (hormone glands being among the most obvious).

> with weighted connections

Probably, though we don't fully understand how synapses work

> performing distributed computation (mainly hierarchical pattern matching)

This is a description of purpose, not form, so it's irrelevant.

> learning from data by gradually updating weights

We have exactly 0 idea how biological neural nets learn at the moment. What we do know for sure is that a single neuron when alone can adjust its behavior based on previous inputs, so the only thing that is really clear is that individual neurons learn as well, it's not just the synapses with their weights which modifies behavior. Even more, non-neuron cells also learn, as is obvious from the complex behaviors of many single-cell organisms, but also some non-neuron cells in multicellular organisms. So potentially, learning in a human is not completely limited to the brain's neural net, but it could include certain other parts of the body (again, glands come to mind).

> using selective attention (and/or recurrence, and/or convolutional filters).

This is completely unknown.

So no, overall, there is almost no similarity between (artificial) neural nets and brains, at least none profound enough that they wouldn't share with a GPU.

[krainboltgreene]

What does this comment add to the discussion?

[robbrown451]

I dunno. My comment complained about the parent comment not adding positively to the discussion. And gave at least a bit of support for that complaint.

Would you have preferred I emulate your style, and complain while providing no support for my complaint?

Ok.

[krainboltgreene]

Being positive is not a requirement of commenting on HN, but you should comment with something that is substantive, so yes I do think you shouldn't have commented at all. Tone policing is cringe.

[JKCalhoun]

I don't like tone-policing in general. But when I opened this post the negative comment we're talking about was the top comment. That's makes me much more sympathetic to someone calling out the cynicism.

[robbrown451]

Exactly what are you doing here then?

But hey I guess I can do this too. How's this? Using cringe as an adjective is cringe.

[krainboltgreene]

> But hey I guess I can do this too.

It sucks, doesn't it?

[jacobsimon]

This is a really weird take. There is such a long history of shared insights between biology and neural network research, and to say they’re unrelated or can’t take inspiration from one another is bizarre.

> The concept of CNNs didn't come from biology

I just opened a survey paper on CNNs and literally the first sentence of the paper reads:

> “Convolutional Neural Network (CNN) is a well-known deep learning architecture inspired by the natural visual perception mechanism of the living creatures. In 1959, Hubel & Wiesel [1] found that cells in animal visual cortex are responsible for detecting light in receptive fields. Inspired by this discovery…”

Source: https://arxiv.org/pdf/1512.07108.pdf%C3%A3%E2%82%AC%E2%80%9A

[jakobson14]

That's later backfill, a retroactive change to give a manufactured "biological" origin story. Whether they're real or not, researchers love a good "we took this from nature, isn't nature wonderful!" explanation.

The C in CNN isn't "Convolution" for no reason. It came from work with convolutional filters (yay Sobel kernels!) which at it's height became filter banks and gabor filters and so on before neural networks pretty much killed off handcrafted feature development. Every explanation of how CNNs work still falls back to the original convolutional kernel intuition.

[rerdavies]

> The C in CNN isn't "Convolution" for no reason.

The first N in CNN is "Neural" for a reason.

[mjburgess]

Can you explain that reason?

Decision trees are called 'trees' for, more or less, the same reason.

ie., the diagrammed shape of a decision tree looks a little like the branches of a real one.

likewise, in the 50s where diagramming the earliest networks they were aiming to immitate a similar real-world structure.

Better that they had called them 'Variable Activation Networks' or some such, and none of this superstition would have started

[TeMPOraL]

> Better that they had called them 'Variable Activation Networks' or some such

But that's the thing: they didn't. Instead, they called them "neural networks". It wasn't random.

It feels like part of the field now wants to pretend it was never about how to make a machine think. "No, we're only doing abstract maths, only going on self-contained explorations of CS theory." Yeah, right. That feels like a reaction to the new wave of AI hype in business. Now that the rubes are talking about thinking machines again, better distance themselves from them, lest we be confused for those loonies.

Thing is, the field was always driven in big part by trying to catch up with nature. It took inspiration from neuroscience, much like neuroscience borrowed some language from CS, both for legitimate reasons. A brain is a computer. It's precisely where the CS and neuroscience have an overlap - they're studying the same thing, just from opposite directions. It's just silly to play the "oh my field is better and your field doesn't know shit" game.

> Decision trees are called 'trees' for, more or less, the same reason.

Decision trees are called after the data structure, which is a way to express a mathematical object, which is older than CS and got that name from... who knows, but my money is on "genealogical tree", which itself is called a "tree" because people back then liked to tie everything to trees (symbol of growth) and flowers and cute animals (symbols of making babies).

The field inherited "trees" from the past. "Networks", too. But "neural" - that was a modern analogy the field itself is responsible for.

[jacobsimon]

Yep! Trees, tree structures, tree diagrams have been regularly in use since the 1700s as a way of defining relationships. https://en.wikipedia.org/wiki/Tree_structure

There’s also a pretty large link between the formal representation of language using syntax trees, which was being formalized by linguists and by programming language developers around the same time: https://en.wikipedia.org/wiki/Formal_language?wprov=sfti1

[dartos]

You can use that argument for anything you disagree with. Do you have a source or anything?

[jakobson14]

Have a read through the first paper describing a convolutional neural network, from 1998: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf

There's absolutely no mention of biological inspiration whatsoever. At the same time, one can point to a long and rich history of convolutional filters being used in signal processing. And then there's the name, Convolutional Neural Network. The entire concept of a CNN is framed as a series of learned filters.

[fastball]

That is definitely not the first paper describing a CNN. That is not even the first paper by Le Cun describing CNNs (he was already on them as early as 1989[1]).

Regardless, Le Cun is not the first to describe CNNs, merely one of the first to use them for OCR (specifically for hand-written text).

The first neural network arch to use convolutions instead of matmuls was this[2], from the year of our lord 1988. This in turn is based on Fukushima's "neocognitron"[3] (1980), which is based on the visual cortex of felines (from work done by Hubel and Wiesel in the 50s/60s).

I guess it is not super surprising you might be confused – Le Cun seems a bit more reticent than average to cite the work he's building on top of, and when he does it is frequently in reference to his own prior work. So if that is where you're getting your picture of artificial neural network history, your skewed perception makes sense.

[1] https://ieeexplore.ieee.org/abstract/document/41400

[2] https://proceedings.neurips.cc/paper/1987/file/98f1370821019...

[3] https://www.cs.princeton.edu/courses/archive/spr08/cos598B/R...

[tudorw]

Thanks, I was looking for something to do with early work and saccades, didn't find that, but found this;

"The most influential of these early discussions was probably the 1943 paper of Warren McCulloch and Walter Pitts in which activity in neuronal* networks was identified with the operations of the propositional calculus. Actual simulations of recognition automata based on networks were carried out by Frank Rosenblatt before 1958 but the theoretical limitations of his "perceptrons" were soon pointed out by Marvin Minsky and Seymour Papert"

excerpt from a 1998 paper, "Real Brains and Artificial Intelligence" (https://www.jstor.org/stable/20025142)

"Walter Harry Pitts, Jr. (23 April 1923 – 14 May 1969) was an American logician who worked in the field of computational neuroscience.[1]"

'https://en.wikipedia.org/wiki/Walter_Pitts'

[jacobsimon]

I don't know why I'm still responding to this thread 24 hours later, but just thought I'd add this tweet from Le Cun: "Neuroscience greatly influenced me (there is a direct line from Hubel & Wiesel to ConvNets) and Geoff Hinton. And the whole idea of neural nets and learning by adjusting synaptic weights clearly comes from neuroscience."

https://x.com/ylecun/status/1583872918634655744?s=20

[jacobsimon]

Surely you are trolling me now. There is a very clear biological inspiration mentioned in this paper: they literally define a CNN as having “receptive fields” and then they cite the same Hubel & Wiesel research mentioned before multiple times. LeCun mentions their research in papers even earlier in the 80s as well, during which they were awarded the Nobel prize for their research on the visual system. Of course there is also a lot of computational and mathematical research that was ongoing simultaneously, but to say that there is “no inspiration whatsoever” is pretty far from the truth.

[jacobsimon]

Plus: https://x.com/ylecun/status/1583872918634655744?s=20

[albertzeyer]

Some time around mid 1995 until basically now, it became out of fashion to explain your motivations of some new modeling as inspired from biology, as that was often handwaving with only little understanding of the actual neuroscience. So that is why people stopped writing that in papers. Just let the actual performance numbers speak for themselves. Either you get good performance, then it doesn't really matter where this was inspired from, or it does not work well, then it also does not matter where this was inspired from. In machine learning, it mostly matters whether it works well or not.

[readthenotes1]

That's funny. I had a book on "neural nets" in the 1980s, and it mentioned the analog to brain neurons.

[two_in_one]

While I agree with this emotional post there is one nuance. Neural networks aren't intelligent, brain is. And that's where we want to be. Checking gradients and studying filters can get us only this far. So, using brain as inspiration looks like a good option. There are other, but nobody knows where next breakthrough will be. Like nobody knew five years back that transformers are so powerful. My guess next step to AGI will be a complex modular multi-modal system. With hierarchy, workers and controllers, complex signals.. Sound familiar? Brain is sort of it. This is need for embodied AI, obviously. But, interesting thing, it's needed even for body-less AGI too. I.e. AGI is not a big calculator (!), it's more like real-time system. One reason is that full search is impossible. So, in many cases requests will be like 'give the best answer you can find in 4 seconds'. 'and keep looking'. So far we have only real-time dumb robots and NN big calculators. And brains, of course.

[dilawar]

> previously: comparing brains and "electronic computers")

Before that: comparing brain with hydraulic machines. There has been tendency to compare brain with most complex machine known to us at that particular time.

"Descartes was impressed by the hydraulic figures in the royal gardens, and developed a hydraulic theory of the action of the brain. We have since had telephone theories, electrical field theories, and now theories based on computing machines… . We are more likely to find out how the brain works by studying the brain itself, and the phenomenon of behavior, than by indulging in far-fetched physical analogies." -- Karl Lashley 1951

[bondarchuk]

Electronic computers, artificial neural networks, hydraulic machines, clockworks etc... are all computationally equivalent to the brain. Anyone making such comparisons is grasping at the fact that the brain can be understood computationally. To complain that there are no pressure-driven pistons, rotating gears or whatever in the brain is missing the point of the analogy, IMHO, which is: all these systems perform computation on top of a physical substrate, and what we actually (should) care about is the computation itself and not the mechanical workings of the substrate.

[mjburgess]

I cannot agree enough with Karl here. What is the brain? An organic system with deep roots in the organic body, with deep causal connections with its environment.

There's little sense in ignoring the whole basic mode of operation, physics, chemistry and biology of the brain in order to analogise it to another system without any of those properties.

This, at best, provides a set of inspirations for engineers -- it does nothing for science.

[TeMPOraL]

> There's little sense in ignoring the whole basic mode of operation, physics, chemistry and biology of the brain in order to analogise it to another system without any of those properties.

Sure there is. People had a feel for it back in "clockworks" times, nowadays we have a much better grasp because of progress of physics and math, particularly CS - mode of operation is an implementation detail. Whatever the mode, once you understand the behavior enough to model it in computational terms, you can implement it in anything you like - gears and levers, pistons, water flowing between buckets, electrons in silicon, photons going through lenses, photons diffusing through metamaterials, sound waves diffusing through metamaterials - and yes, also via a person locked in a room full of books telling them what to draw in response to a drawing they receive, and also via a billion kids following a game to the letter, via corporate bureaucracy, via board game rules, etc.

Substrate. Does. Not. Matter.

The only thing limiting your choice here is practical one. Humanity is getting a good mileage out of electrons in silicon, so that's the way to go for now. Gears would work too, they're just too annoying to handle at scale.

Of course, today we don't have a full understanding of biological substrate - we can't model it fully in terms of computation, because it's a piece of spontaneously evolved nanotech and we barely begun being able to observe things at those scales. We have a lot of studying in front of us - but this is about learning how the gooey stuff ticks, what does it compute and how. But it's not about some new dimension of computation.

[mjburgess]

> Substrate. Does. Not. Matter.

It only doesnt matter for counting a system as implementing a pure algorithm, ie., one with no device access. This is an irrelevant theoretical curiosity.

Electronic computers are useful because they're electronic -- they can power devices, and modulate devices using that power. This cannot be done with wood, or most anything else.

"Substrate doesnt matter" is, as a scientific doctrine pseudoscience, and as a philosophical one, theological.

The causal properties of matter are essential to any really-existing system. Non-causal, purely formal properties of systems which can be modelled as functions from the naturals to the naturals (ie., those which are computable) are useless.

[TeMPOraL]

> Electronic computers are useful because they're electronic -- they can power devices, and modulate devices using that power. This cannot be done with wood, or most anything else.

On the contrary. That's an implementation detail. You can "power devices, and modulate devices" by having a clockwork computer with transducers at the I/O boundary, converting between electricity and mechanical energy at the edge. It would work exactly like a fully electronic computer, if built to implement the same abstract computations - and as long as you use it within its operational envelope[0], you wouldn't be able to tell the difference (except for the ticking noise).

> The causal properties of matter are essential to any really-existing system. Non-causal, purely formal properties of systems which can be modelled as functions from the naturals to the naturals (ie., those which are computable) are useless.

Yes and no. Of course the causal properties of matter... matter. But the breakthrough in understanding, that came with development of computer science and information theory, is that you can take the "non-casual, purely formal" mathematical models of computation, and define some bounds on them (no infinite tapes), you can then use the real-world matter to construct a physical system following that mathematical model within the bounds, and any such system is equivalent to any other one, within those bounds. The choice of what to use for actual implementation is done on practical grounds - i.e. engineering constraints and economics.

It's how my comment reached your screen, despite being sent through some combination of electrons in wires, photons down a glass fibre, radio signals at various frequencies - hell, maybe even audio signals through the air, or printouts carried by pidgeons[1]. Computer networks are a living proof that substrate doesn't matter - as long as you stick to the abstract models and bounds described in the specs for the first three layers of ISO/OSI model, you can hook up absolutely anything whatsoever to the Internet and run TCP/IP over it, and it will work.

I bet there's at least one node on the Internet somewhere whose substantial compute is done in a purely mechanical fashion. And even if not, it could be done if someone wanted - figuring out how to implement a minimal TCP/IP stack using gears and switches is something a computer can do for you, because it's literally just a case of cross-compilation.

--

[0] - As opposed to e.g. plugging 230V AC to its GPIO port; the failure modes will be different, but that has no bearing on either machine being equivalent within the operational bounds they were designed for.

[1] - https://datatracker.ietf.org/doc/html/rfc1149

[mjburgess]

> matter to construct a physical system following that mathematical model within the bounds, and any such system is equivalent to any other one, within those bounds

No. This wasnt discovered.

Nearly every physical system is implementing nearly every pure algorithm, ie., every computable function.

The particles of gas in the air in my room form a neural network, with the right choice of activation function.

Turing-equivalence is a property of formal models with no spatio-temporal properteis. Physical systems are not equivalent because they both implement a pure algorithm

Pure algorithms are useless, and of interest only in very abstract csci. All actual algorithms, when specified, have massive non-computational holes in them called 'i/o', device access etc.

If your two systems of cogs wants to communiate over a network of cogs, the Send() 'function' (which is not a function!) has to have a highly specific causal semantics which cannot be specified computationally.

These systems only have 'equivalent functions', as seen from a human point-of-view, if their non-computational parts serve equivalent functions. This has nothing to do with any pure algorithm.

You cannot implement a web browser on 'gears' in any useful sense, in any sense in which the partices of their air arent already implementing the web browser. That a physical system can-be-so-described is irrelevant.

Computers are useful not because theyre computers. Theyre useful because they are electrical devices whose physical state can be modulated with hyper-fine detail by macroscope devices (eg., keyboards). We have rigged a system of electrical signals to immitate a formal programming langauge -- but this is an illusion.

Reduce the system down to just want can be specified formally, and it disappears.

[ben_w]

I mildly disagree (although your final conclusion is correct: it indeed does nothing for science).

The deepest fundamental structures in the brain[0] are quantum fields, which are also the deepest fundamental structures in everything else.

There is no known quantum field of "soul" or "intelligence".

The right abstraction is higher, and could still be a whole lot of things; but as maths can be implemented in logic, which can be implemented in electronics or clockwork or hydraulics, it doesn't matter what analogy is used — and my mild disagreement here is that such inspiration has been useful and gotten us this far.

[0] that we know of

[mjburgess]

The process of evolution acts on organic systems, it doesn't act on quantum fields.

I appreciate there's some (imv strange) sense of 'intelligence' where 'finding the right puzzle piece' counts. I cannot fathom why we care about such a notion, and it seems to have almost nothing to do with what we do care about re 'intelligence'.

We care about that thing animals do, that thing which some do better than others. That thing which evolution brought about for (rapid) adaptive fitness to one's environment.

'Everything else is stamp collecting'

We already have a perfectly good understanding of puzzles and their solutions -- animals are their inventors

Intelligence isnt in the solution to a puzzle it's in its design, and especially, in what one does when one cannot solve it -- ie., how one adapts

The csci view of 'intelligence' is an act of self-aggrandising, it turns out to be: csci!

This is none-sense.

[ben_w]

We can simulate evolution in a computer, and this is used as a form of AI directly.

That said, the way you're using biological evolution in your comment sounds as much like a strange analogy as all of the others: we may have some genetically programmed responses to snakes (bad) and potential mates (good), but we can also say that a loss of hydraulic pressure in our brain is a stroke, and use electrical signals to both read from and write to the brain.

What we evolved to think, while interesting from a social perspective, seems to me like the least interesting part of our brains from an AI perspective — it's the bit that looks like a hard-coded computer program, not learning, on the scale of a human life and seen from within.

[mjburgess]

i'm referring to evolution as the process by which animals were built

if aliens had come down and given us laptops, rather we invented digital machines, then likewise i'd be talking about the relevant materials science, physics etc.

reverse engineering a laptop to figure out how it works would require extremely little computer science, and 'only at the end'

the reason digital computers are interesting and useful is that they route electricity around devices which are designed to be responsive to one another. the patterns of activation, as managed by the CPU, are weakly describable by abstract algorithms like sorting

starting with a laptop, and no further information, we'd be 100(s)+ years of research away from needing to understand that CPUs were implementing a sorting algorithm

and importantly, that it is doing so has almost nothing to do with the value of the device -- which lies in its ability to provide 'dynamical power and modulation of operation' using electricity

we're in the same situation with animals and people think that, what, understanding gradient descent or backprop is helpful? this is just some csci bs

[spindle]

And also comparing brains to clockwork.

[crustacean111]

CNNs actually are biologically inspired. The receptive field in a CNN mimics the way that cortical neurons only respond to stimuli in a restricted region of the visual field. Different cortical neurons have receptive fields that partially overlap to cover the whole visual field [1].

[1] - https://en.wikipedia.org/wiki/Convolutional_neural_network

[jakobson14]

You're going to have to dig deeper. The concept of a receptive field goes all the way back to convolutional filters.

It's not surprising that we found out later the brain also uses such a fundamental element of signal theory.

[SubiculumCode]

Oh good. So you do admit that there are useful parallels between signal processing, statistical processing, and the brain.

[vkou]

Sure, and airplanes are inspired by birds. That doesn't mean that detailed studies of the Boeing 747 are going to unlock a lot of hitherto unknown mysteries of heron behaviour.

[jacobsimon]

I mean, I know you’re just providing an analogy, but people are still studying the physics of bird flight and we’re nowhere close to building machines yet that can maneuver the way birds can. https://www.quantamagazine.org/geometric-analysis-reveals-ho...

[ben_w]

I could believe "we have more to learn", but not "we're nowhere close":

https://youtu.be/w6VLzKACnS8?si=DZgOPuBRG4Vt98su

[jacobsimon]

TIL

[wslh]

Only an observer of the topic but I think it is good to review Koch's book about the real complexity of a single neuron [1].

[1] https://www.amazon.com/Biophysics-Computation-Information-Co...

[mrstone]

A neurologist is a medical doctor. Neuroscientists are the PhDs who do the actual research.

[blovescoffee]

Dude. What holy and special work do you do? There's nothing dumb or dull in searching for analogous structure between two effective machines, neither of which we understand.

[hliyan]

"brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong"

Please don't claim things the author didn't. What I read was "ergo (artificial) neural networks may be missing a trick"

[b33j0r]

Agreed, but I do also think that order emerged from chaos. It’s an easy claim when order is defined by itself!

But in reality, we’re equipped exactly to exist, and we still wonder why in a backwards way, even with education (guilty!)

AI is the task of playing God like toddlers at recess, and LLMs the tower of babel. I still wanna play, it’s fun

[bjourne]

First, I wonder how you got access to the article? It is behind a paywall and not yet uploaded to the sites I usually find paywalled articles on.

Second, there is no need to compare brains to neural networks because brains are neural networks. Neurons form vertices and axons edges connecting the aforementioned. What you are perhaps thinking of are artificial neural networks - most of which are very dissimilar to brains. But even then you are wrong. Artificial Izhikevich and Hodgkin-Huxley neural networks attempts to closely mimic the behavior of real neurons.

While deep, hierarchical artificial neural networks have been more successful than biologically plausible ones, that may be because the technology isn't ready yet. After all, the perceptron was invented in the 1950's but didn't become prominent until the 2010's (or so). Perhaps we need new memories that better map to (real) neural network topologies, or perhaps 3d chips that can pack transistors in the same way brains pack neurons.

[mjan22640]

A neuron is analogous to a 3d integrated circuit rather to a transistor. A molecule acts like a transistor https://medium.com/the-physics-arxiv-blog/the-origin-of-life...

Changes in mechanical pressure, electric field, other molecules attachment, photon absorption, can control the conductivity.

Organic semiconductors designed to fit like lego bricks to naturally build the desired structure are IMHO the way to go to produce 3d circuits, rather than layered silicone litography.

[ben_w]

> silicone

I've seen this particular mistake a lot recently. New and exciting auto-corrupt from the latest version of iOS?

Given that our brains rewire themselves live, which ANNs can only do by being excessively connected and updating weights to/from zero, silicone (I'm thinking mainly the oil form) may be a better inspiration than lego.

https://en.wikipedia.org/wiki/Silicone

[mjan22640]

Yes, puzzle pieces would be more accurate than lego.

The bonds that silicone forms do not AFAIK allow as rich variety of polymers as carbon.

[ben_w]

Silicone, with the e on the end, is one of the main polymers.

[SubiculumCode]

If you read this article, I think most would understand that it is primarily aimed at other neuroscientists, and only using ML structures an an analogy only, and I think a somewhat useful one to boot. The real point of the article was to propose a general hierarchy for how information flows in the brain, to emphasize the importance of subcortical brain even in higher order cognition, and proposes how simultaneous processing of multiple levels of representation can inform action and thought.

As a developmental neuroscientist, I found the article insightful and thought provoking. Further, it is quite consistent with major hypotheses in psychology, how the hippocampus works (a subcortical structure) and combines information into memories: See fuzzy trace theory [1], for example.

Your dismissive tone is unappreciated, ill-informed, and crass.

[1] https://en.wikipedia.org/wiki/Fuzzy-trace_theory

[visitor4711]

fully agree

[radarsat1]

> every time some neurologist tried to compare brains to neural networks

Value of this comment aside, it kind of makes me chuckle how casually it (and other comments in this thread) just drops the word "artificial" from neural networks here, specifically when comparing with neurology. The irony is funny. Like, somehow we've forgotten why we call them that in the first place, exactly when talking about the thing that inspired the approach.

[NoToP]

I disagree profoundly.

There are things the brain does we have not yet been able to reproduce with a neural network, or to the extent we have seemingly with excessive resources of training and network size. Therefore there is some salient feature of neurology which has been overlooked. I don't think it is necessary to mimic biology down to the exact function of real neurons, but there must in fact be something we are neglecting to mimic.

[ben_w]

Possibly, but it may also be that we're training them wrong.

"Book smart, not street smart" (to use a catchphrase) would apply perfectly to GPT models: brain the size of a rodent's, with 50,000 year's experience of reading Reddit, Wikipedia, and StackOverflow, but no "real life" experiences of its own.

[andromaton]

Books and articles I was reading in the 80s (eg Minsky and Papert, Byte magazine) were referring to Rosenblatt and retinas.

[nathias]

Metaphores and analogies are important tools of thinking, even in science, some bear fruits some lead to errors, but we can't know in advance.

[peyton]

I dunno, failure seems okay. Wouldn’t expect a better paradigm to beat SOTA at first. It’s totally plausible that neurons use eg. transposons in a way we don’t yet have the instrument resolution to characterize, which would suggest that you don’t need 1000 layers, but a lookup table or something.

[svara]

Doesn't know what a neurologist is, knows they do shit work.

[__loam]

As a biomedical engineer who went into software, thank you for this comment lol. So tired of rehashing this.