[cma]

Didn't Schmidhuber's lab predate all of those in their inventions except ReLU (which I've heard is now out of favor as compute outpaces memory bandwidth and other activation functions have better properties).

[chubot]

Yeah I should have mentioned him too as evidence of the point!