[dnautics]

> I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity.

At that point, you are probably better off just building circuits with "power this wire to ReLU at the end", which is not very many extra transistors.