Hacker News new | ask | show | jobs
by erAck 2897 days ago
As you referenced that bug and there also gave an example involving fractions and =0.1+0.2 you would certainly also agree that the binary floating point representation of =0.1+0.2-0.3 is not 0.0, still Excel displays 0 as result (and so does Calc) instead of 5.551115123125783E-17 because users expect that.

Your bit-level accuracy approach isn't as simple when it comes to user experience and Excel compatibility.
1 comments
erAck 2897 days ago
Regarding your fraction example there:

=RAWSUBTRACT(0.1,-0.2,1/3) => -0.033333333333333

=RAWSUBTRACT(0.1,-0.2,2/7) => 0.014285714285714

So which one is closer to 0.3?

link