Hm, perhaps the answer is a subtle definition of "rounding error".
A rounding error is when you have a number, lets say 0.75, and due to rounding it is recorded as 1.00. The "rounding error" is 0.25.
An alternative to rounding to 1.00 would be to have a mechanism which says "the value is between 0.50 and 1.50". This way, there is no actual rounding, as it doesn't commit to a rounded value, so there is technically no "rounding error".
A neat advantage of recording an interval rather than rounding is that the "error" is preserved in the data through arithmetic, so if there is some following code that runs x * 100, a rounding mechanism would say "the value is 100" whereas an interval mechanism would say "the value is between 50 and 150". Then, if the user only looks at the output, it will be clear that there is a wide error range and something needs to be fixed, rather than the output indicating a precise answer when really it suffers from significant rounding errors.
That claim can only be made with respect to using a Kulisch accumulator (what he calls the quire) for accumulation.
This can represent the exact sum of any number of floating point values (or products of floating point values) with the intermediate operations preserved at infinite precision, up to a single rounding at the end.
e.g., for computing an inner product of vectors \sum_i x_i y_i with x_i and y_i in floating point, a fused multiply-add operation (as on today's computers) would perform something like:
where what is performed within the rounding function r() is done to infinite precision and a single rounding at the end. This is also only true if you are accumulating into the same value, and not splitting this operation across multiple values, otherwise there would be additional rounding steps performed.
Using a Kulisch accumulator the result would be:
r(\sum_i x_i y_i)
similarly also only preserving precision if everything is done in a Kulisch accumulator.
You can add a Kulisch accumulator to IEEE floating point or any other floating point as well. In fact the second? ever computer with floating point (the Zuse Z3) performed FP addition using a similar accumulator, but sums-of-products probably didn't preserve the multiplied value though.
Yes, this is the question I come to. The article betrays its own claim...
> Gustafson said that for training, 32-bit floating point is overkill and in some cases doesn’t even perform as well as the smaller 16-bit posits
If posits have a bit number, then they are not of variable accuracy. This is simply sloppy explaining.
At some point, rounding has to happen (call it an interval if you want, but you're not being helpful). As a programmer, I WANT rounding to happen. I depend on it. You don't break out the floats unless you are prepared for some information to be lost. If you try to never lose any information, then you wind up with a monstrosity like Maple and Mathematica in which all steps need to be curated by hand to occasionally reduce the giant glob of fractions and implicit solves into something concrete, but imperfect.
One form of rounding error that comes up in floating point conversations has to do with data aggregation. If the average value of a series is 3.5 but after rounding the average drops to 3.4, then your rounding has lost some fidelity, and this upsets some people. This counts as a rounding error to them.
A rounding error is when you have a number, lets say 0.75, and due to rounding it is recorded as 1.00. The "rounding error" is 0.25.
An alternative to rounding to 1.00 would be to have a mechanism which says "the value is between 0.50 and 1.50". This way, there is no actual rounding, as it doesn't commit to a rounded value, so there is technically no "rounding error".
A neat advantage of recording an interval rather than rounding is that the "error" is preserved in the data through arithmetic, so if there is some following code that runs x * 100, a rounding mechanism would say "the value is 100" whereas an interval mechanism would say "the value is between 50 and 150". Then, if the user only looks at the output, it will be clear that there is a wide error range and something needs to be fixed, rather than the output indicating a precise answer when really it suffers from significant rounding errors.