[Schlagbohrer]

Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)

[rahen]

The first AI winter was largely triggered by Minsky in a book he published in 1969, which mathematically proved that single-layer perceptrons couldn't solve non-linear problems. Favorite quote: "Our intuitive judgment is that the extension [to multilayer systems] is sterile."

Yet we had the computational power to run backpropagation in the 1960s and small Transformers in the 1970s (I'm the author of both):

https://github.com/dbrll/Xortran (backprop on IBM 1130, 60s)

https://github.com/dbrll/ATTN-11 (Transformer on PDP-11, 70s)

What was missing wasn't the raw processing power, but the ideas and algorithms themselves. Because funding and research were completely discouraged during the AI winter, neural networks research was left dormant and we lost two decades.

[WarmWash]

I wonder had we invented transformer architecture back in the 70's or 80's, if the pace of hardware innovation would have naturally slowed AI progression, and given humans decades to slowly adapt, rather than the current tidal wave (that seems to grow in size daily) bearing down on us.

[Schlagbohrer]

This is a fascinating discussion, thank you both.

I'll add this to my casual mental list of "Technologies that could have been developed generations earlier but through random chance, simply weren't."

[mr_toad]

> why there wasn't more progress on neural nets being used for many things prior to the 21st century

They were simply too computationally expensive to train for the limited things they could do. It wasn’t until we had the ability to train large neural networks on commodity hardware that things really took off.

[j_bum]

This doc on Ilya Sutskever & Geoffrey Hinton gives a great background on the progression of deep learning over the past decades [0].

Tl;dr - compute was the bottleneck.

I am not associated with this channel/video, just love it. I’ve shared it here before.

[0] https://youtu.be/glWvwvhZkQ8?si=XjcwWWy43305tl6O