Hacker News new | ask | show | jobs
by p1esk 836 days ago
This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.
2 comments

Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part.

I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?

But this group claims to have demonstrated a way to use SNNs to run LLMs effectively and with vastly less energy usage.
If OpenAI or DeepMind makes such claim I'd pay attention. Otherwise it's always some (usually hw) guys trying to get a grant, or even just publish a paper.

p.s. People interested in biologically inspired data processing algorithms should look at Numenta's papers (earlier ones, because recently they switched to regular deep learning), and especially learn their justification for not using spikes.

They switched? Why? There goes their whole raison d'etre!
Spiking neural networks are not software, they are usually built directly into silicon chips because they are using pulse timing to encode information instead of multiple bits. The problem is that training them is difficult because they operate over time, not that they don't work. As of now, scaling training infrastructure is more important than theoretical power efficiency.