[conistonwater]

If you are interested in writing accurate numerical algorithms, as further reading I highly recommend reading "Accuracy and Stability of Numerical Algorithms" by Nick Higham, which is a very good comprehensive book. Like this article says, floating-point arithmetic is often thought of as mysterious, but it really isn't: it just obeys its own precisely specified rules. I think it's very good to dispel the mystery, so I really like this article.

Another way to do the same type of error analysis of numerical algorithms is to use a parameter sensitivity library. In the end, numerical stability can be thought of as the sensitivity of the final answer to separate independent changes in each intermediate value obtained in the computation of the answer. This is sometimes overlooked, I think, and it's quite easy to implement as well.

I have one quibble. They say "Use (x-y)(x+y) instead of x^2-y^2". This is true, but (in my opinion) misleading: the function (x,y) -> x^2-y^2 is itself ill-conditioned when x is near y, and using a more numerically stable algorithm to evaluate an ill-conditioned function will not improve the function's condition. If x and y are close to each other, the inaccuracy introduced by using x^2-y^2 is close to the inaccuracy due to using approximate values for x and y. So it you have x^2-y^2, I think the first thing you should worry about is whether your computation is ill-conditioned, and only then worry whether it's numerically stable. I say this because I sometimes see people try to evaluate x^2-1, notice it is inaccurate for x near 1, and try to replace x^2-1 with (x-1)*(x+1), which is still inaccurate.

[dmlorenzetti]

The first thing you should worry about is whether your computation is ill-conditioned, and only then worry whether it's numerically stable.

OK, but how does one apply this advice in practice?

The radiative exchange between two surfaces goes as x^4-y^4 where x and y are the surface temperatures. I can worry about whether that's ill-conditioned all I want, but at the end of the day I'm further ahead evaluating (x-y)(x+y)(x^2+y^2) than the original expression.

Similarly, if I need to find x^2-1, what recourse do I have? Maybe there's some reduction that makes the computation more stable. But normally I would expect to discover such a reduction by turning x^2-1 into (x-1)(x+1), and then seeing if one of those factors cancels out somewhere. So it still seems like a good first step.

[conistonwater]

I don't know about your radiative exchange example, but it is often possible to reparametrize the problem. For example, sometimes you can replace the parameters (x,y) with (s,t)=(x-y, y), which would get you further: x^2-y^2 = s (s+2t), which is stable at s=0, and if x>y>0, then it's stable everywhere. This can shift the region of instability to somewhere far from typical parameter values. Otherwise, yes, this isn't solvable.

[stephencanon]

The correct way to evaluate x^2 - 1 is by using fma(x,x,-1). Now that Intel and AMD have finally made FMA available in hardware on commodity parts (better late than never!), it's realistic to start using this much more freely.

[conistonwater]

That won't do much for x near 1, because the function x^2-1 is itself ill-conditioned. In other words, it is relevant that the floating-point value of x is itself only an approximation to some true value of x. So computing x^2-1 exactly for a given floating-point value of x does not give a good approximation to the true value of x^2-1. This is a mathematical property of the function x^2-1, and cannot be fixed with any algorithm. This is basically why I consider the example x^2-1 => (x-y)(x+y) misleading.

[stephencanon]

There are multiple ways to analyze computations; condition number is one of them. It is relevant when inputs are assumed to be approximations to some hidden "correct" value. However, this assumption is not always warranted when dealing with floating-point numbers; sometimes, we would instead like to analyze errors under the assumption that the inputs are exact values. When this is the case, fma(x,x,-1) produces a correctly rounded result, whereas (x-1)(x+1) does not, and the ill-conditioning of the function x^2 - 1 does not enter into the analysis anywhere.

If you are used to working with complex computations (most of what is usually referred to as numerical analysis, for example), you likely do not find yourself in the latter situation very often. However, for those of us who work on low-level details of floating-point on a regular basis, the latter analysis is often critical.

[conistonwater]

Okay, I agree, that is completely reasonable. I do usually work with fairly complex algorithms, so I care about individual operations less than about the final result.

[acqq]

The expression won't turn to be good conditioned only by using fma. You actually missed the point of conistonwater's post.

[stephencanon]

Condition number only matters if you assume that the inputs are perturbed by some error. Under the assumption that inputs are exact (which is appropriate to make for some uses), it does not enter into the analysis.

[dalke]

"Use (x-y)(x+y)" is part of the "folk wisdom." It's between "We even know some folk wisdom intended to avoid the dangers." and "or is it increasing magnitude?", where the last bit shows that perhaps these aren't the best solutions.

[conistonwater]

No, later in the article they recommend using it, saying that it decreases floating-point errors.

[dalke]

Ahh! Yes, in the box on p94 titled "About That Folk Wisdom". Thanks for the correction!