[dnautics]

Agreed. For neural networks. I would argue the opposite is true, you should just have a 16bit float that casts really large values to infinity silently without throwing errors, with a logistic lookup that maps "inf" to +/- 1... A mathematically incorrect float is operationally superior to the correct one.

[methodOverdrive]

Bonus points if values near 0 are treated as 0 (encourages sparsity!)