[wrsh07]

Sort of, yes. But if the existing thing were "the cheapest", quantization wouldn't exist.

It depends on what your constraint is! So if you're memory constrained (or don't have a GPU), a bunch of 1 bit atoms with operations that are very fast on CPU might be better

I haven't thought very deeply about whether it's provably faster to do gradient descent on 32 bits vs 8, but it probably always is. What's the next step to speed up training?