Hacker News new | ask | show | jobs
by Conlectus 753 days ago
Isn’t ReLU an element of neuroscience that was incorporated into machine learning to great success?
1 comments

Not really, no. That's motivated by not getting impractically small gradients on the plateaus and spoiling the optimization properties when used for deep ANNs. The sigmoids it replaced had a bit more neuroscience inspiration, but so oversimplified it's just barely.