Hacker News new | ask | show | jobs
by slver 1808 days ago
Excel rounds doubles to 15 digits for display and comparison. The exact precision of doubles is something like 15.6 digits, those remaining 0.6 digits causing some of those examples floating (heh) around.
1 comments

A lot of these edge cases are about theoretical concerns like "how many digits we need in decimal to represent an exact IEEE binary float".

In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness.

In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.