[dannymi]

>These are placed on the outputs of a neuron and are one of those weird mathematical hacks where I can't really explain why it's needed,

Because when you compose linear functions you get linear functions. So having linear everything is a waste of all layers but one.

In order for this not to happen, you need nonlinearity.

[mike_hearn]

Thanks!