[Hyperbolic]

I am always skeptical of people trying to "emulate" the human brain in machine learning. We currently do not have the tools to accurately record and analyze the dynamics of networks of neurons in the brain, and any group that claims to advance ML with knowledge of the cortex is spouting bullshit. Modern advances in ML are driven by great engineering, not biological insight.

[shahbaby]

Even DeepMind says that future advances will come from neuroscience.

https://www.theverge.com/2017/7/19/15998610/ai-neuroscience-...

In the field of AI, our "great engineering" is not even a worthy comparison to what nature has achieved. Maybe there are a few more things to learn from it.

Also we've been studying the neocortex for a long time and have learned a lot more about it than most people realize.

[ogrisel]

While Demis Hassabis and probably some DeepMind researchers use some understandings from the neuroscience literature as an inspiration for their work, I am pretty sure that a majority of DeepMind researchers would rather use a combination of mathematics and trial and error experiments to build an intuition to guide the design of their next iteration of intelligent learning systems.

[shahbaby]

The point is that when mainstream ML begins to realize that there are limitations to the current cartoonish representations of neural networks, they then go back to the biology to see what they're missing.

However there are other companies, like Numenta, which realized decades ago that the current techniques will not be sufficient for general intelligence.

Numenta is not trying to emulate the brain like the Human Brain Project, they are aiming to learn the principles behind the neocortex and replicate it in software.

Again, I don't think most people know enough about the neocortex because if they did, we probably wouldn't be so quick to discard the only real example of intelligence we have.

[ganeshkrishnan]

>there are limitations to the current cartoonish representations of neural networks

How? on a broad level Deep Learning is the same as natural neural network. Signal in and then neuron decides to fire a signal out. The algorithms inside is what differentiates a human from a machine. As long as the algorithm can make intelligent decisions who cares how the human algorithm works. We are not trying to build a human brain, we are trying to build a better than human brain

[ogrisel]

To be more factual: a majority of papers published by researchers at DeepMind do not cite any result from the neuroscience literature in their bibliography. Instead they cite other papers from the Machine Learning community.

[Houshalter]

I recall a quote from some neuroscientist. That whenever he hears people say "we know nothing about the human brain", he wants to smack them with a 900 page neuroscience textbook.

I think the biggest issue is these domains are isolated and don't talk to each other. It's not that ML researchers couldn't be inspired by neuroscience research. They just don't know any.

I talked to a researcher outside of the mainstream who was obsessed with biologically plausible models. He got good results, but not SOTA.

However his main argument was that his methods were much faster and more data efficient than standard practice. E.g. they did online learning and didn't suffer from catastrophic forgetting. Didn't require supervision and labelled data.

Standard methods are optimized towards getting the most accuracy on benchmarks, and not necessarily under realistic conditions. Real brains don't get to save huge dataset and iterate over them later. They need to learn in real time and without forgetting previously learned knowledge. Given just a stream of unlabeled data. ANNs can't do this at all. Some biologically inspired models claim to be able to do this well.

[eli_gottlieb]

>Real brains don't get to save huge dataset and iterate over them later. They need to learn in real time and without forgetting previously learned knowledge. Given just a stream of unlabeled data.

It's more than just that. Real brains need to optimize their internal data for action. The ultimate test of whether you've represented the world correctly is: but can you do stuff? Can you control an inverted pendulum (to pick a task) while constrained to have your representation updates be Lipschitz functions, with a Lipschitz constant based on your actuators' state (ie: speed, angle, force, etc)?

Note the trick here! Your representation doesn't have to be reconstructive itself (allowing for you to conditionally simulate only Lipschitz transformations), but the updates you perform on that transformation from sensory reafferant signals do need to change only at a bounded rate, because the physics of the thing you're moving actually have that property.

[tomp]

> I talked to a researcher outside of the mainstream who was obsessed with biologically plausible models. He got good results, but not SOTA.

Can you expand on this? Do you have some resources that describe (something similar to) what he did?

[Houshalter]

It was pretty similar to Numentas stuff and hierarchical temporal memory.

[jgalt212]

Numenta has been working on this strategy since 2005. Has anyone identified any of their products in wide use?

https://en.wikipedia.org/wiki/Jeff_Hawkins#Numenta

[hacker_9]

Hey we fully mapped a worm's ~300 neurons [1], now we just need to keep scaling up until we get to the billions that the human brain is made up of.

[1] http://www.openworm.org/

[eli_gottlieb]

>Hey we fully mapped a worm's ~300 neurons [1]

And functionally, what did that tell us?

[hacker_9]

They allowed the connectome to control a lego robot [1] and it showed some signs of working around obstacles (robot backs up when it encounters a wall). Sounds simple, but now we can say that only 300 neurons are needed to be able to react to the environment in this way, which is a huge stepping stone in understanding.

[1] Video: http://www.smithsonianmag.com/smart-news/weve-put-worms-mind...

[eli_gottlieb]

Ok, but what functional knowledge did we gain? Not "run this neural net, and an interesting thing happens". How are those 300 neurons implementing interesting motor behavior? What behavior are they really implementing? What's the algorithm, what's its goal?

The connectome model only addresses the third and lowest levels of Marr's analysis of a cognitive/biological system.

1. Computational: What does the system aim to do? What problem does it solve?

2. Algorithmic: How does the system solve or approximately solve that problem? How does it accomplish its purpose as a part of the organism?

3. Implementation: How are cells and/or organs put together to implement that solution?

You can have a very accurate picture of (3), and still lack any solid knowledge about (2) or (1). You can also sometimes have a clear picture of (2) and (3), but overgeneralize and wind up with a near-tautological theory for (1) -- that's the accusation being thrown at certain parts of theoretical neuroscience today.

[hacker_9]

You talk like the project is over, there is still a lot to learn and we aren't going to figure out everything immediately. A simulation provides an amazing test bed for ideas though; now you can see it working, you can pause it, change things, do experiments, and so on.