Hacker News new | ask | show | jobs
by macrael 4255 days ago
Does a "typical" neural network not have any storage to speak of? When I've seen examples of neural networks working, it's seemed like they work in cycles in some way, with the states of each "neuron" affecting the state of others. Is that not potentially storage?
2 comments
robert_tweed 4255 days ago
Recurrent neural networks work that way, but typically when anyone says "neural network" they mean a feed-forward network, which is as dumb as a lead pipe. It has no memory other than the trained synapse weights. After training, all it does is transform some input data into output data. It's a pure function. It's also a lot like a matrix transform.

I haven't read this paper fully yet, but it seems to be an attempt at simplifying RNNs by replacing some of the magic internal state, which tends to make them hard to reason about, with a more direct memory architecture.

link
albertzeyer 4254 days ago
Yes, but its finite storage. (You could argue that with infinite float precision, you would could get infinite storage. Some old proofs that RNNs are as powerful as a Turing machine work that way. Of course, in practice, it's very much limited.)

Also, it's somewhat hard to train the network that it can remember much information. (Not sure if much was done to measure this, but from my gut feeling, I would say, a RNN layer with about 500 neurons can be trained with the standard methods to maybe store 10 bits of information effectively.) The problem is that it can become easily very unstable.

The LSTM cell is already somewhat better in this regard. But of course, this is still finite memory then, and you cannot have much more than 500-1000 LSTM cells in one layer, because training becomes too computational expensive then. (You could introduce some bottleneck projection layer as Google did recently, and then get it to maybe 2000 LSTM cells.) Maybe count one LSTM cell as one bit of information (again, this is very much taken out of air).

This is much less memory than what your PC has (which of course also has finite memory, but it's so huge that you can count it as infinite and as powerful as a Turing machine).

link