[Nokinside]

This is one cool hack.

You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transformation matrix. That's why you need alternating linear and nonlinear operations.

I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. Then you have fast DNN network pipeline constructed from just summation and addition.

[darkmighty]

It's also possible to binarize DNNs to use the faster bitwise operations

https://arxiv.org/pdf/1602.02830.pdf

[dnautics]

> I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity.

At that point, you are probably better off just building circuits with "power this wire to ReLU at the end", which is not very many extra transistors.