Hacker News new | ask | show | jobs
by marcandrysco 3800 days ago
This problem is even trickier than that. The core of the algorithm should be robust given different rounding modes, but the values in the lookup table may differ based on the rounding mode. While it may seem odd for the rounding mode to change at run-time, it has happened before: https://blogs.msdn.microsoft.com/oldnewthing/20080703-00/?p=....

I will have to look at this further. It might mean generating lookup table for every rounding mode and using their union. In general, changing the global state of the FPU seems to be a risky proposition.
1 comments
lifthrasiir 3800 days ago
Hmm, just out of curiosity, would it be possible to tweak Grisu to use the same error analysis as Errol? While possibly slower in the modern architecture, it might result in the compact, fast enough and hassle-free algorithm that is good for most [1] purposes.

[1] I have seen some performance-oriented JSON encoders using Grisu2 instead of Grisu3 in favor of performance. They may be willing to switch to Errol3 in spite of such configuration problems...

link
marcandrysco 3800 days ago
That really depends on what you are trying to get out of the algorithm. Using 64-bit integers, I don't think you can turn Grisu2 into an failure-free algorithm. However, you can definitely improve its reliability by handling special casing small integers (as is done in Errol2/Errol3). By my estimates, this should improve Grisu's success rate from ~99.5% to ~99.95% [see math at end]. The remaining 0.05% is far too large to for enumeration and using a lookup.

If you are willing to print numbers that are correct but possibly suboptimal, you can also use Errol2 and modify it to remove checking. This will give you the speed of Errol3 without having a lookup table. I might look at making such an implementation and maybe call it Errol2b.

[math for calculating approximate error rate] With an input floating-point format with p bits of precision and an intermediate format with q-bits of precision: the "chance" of value failing is close to 1 in 2^(q-p). With 2^p values per exponent, you're going to see about 2^(2p-q) errors per exponent. In the case if Grisu, q is 64 and p is 53, for a total of about 2^42 errors. These numbers only give rough estimates, you'd have to verify them manually.

link