Stuff like Kahan summation breaks horribly with -ffast-math, because if you assume that sum is associative the error term, which is ((s+x)-s)-x, simplifies to zero.
Yeah, that's the obvious counterexample, but that's why I said "more often than not". The statement I was questioning was that fast-math is "normally not more correct."