[celegans25]

On gcc 13, the difference in assembly between the min(max()) version and std::clamp is eliminated when I add the -ffast-math flag. I suspect that the two implementations handle one of the arguments being NaN a bit differently.

https://gcc.godbolt.org/z/fGaP6roe9

I see the same behavior on clang 17 as well

https://gcc.godbolt.org/z/6jvnoxWhb

[gumby]

You (celegans25) probably know this but here is a PSA that -ffast-math is really -finaccurate-math. The knowledgeable developer will know when to use it (almost never) while the naive user will have bugs.

[mort96]

What you really should enable is the fun and safe math optimizations, with -funsafe-math-optimizations.

[gumby]

The problem is this causes the compiler to correctly solve your recreational math problems, which isn't actually a much fun as solving them yourself!

[aqfamnzc]

I know almost nothing about compiler flags but I got a laugh out of this even though I still don't know if you're joking or not. Edit: Just read it again and now I understand the joke. Haha

[kevincox]

To others `-f` is a common prefix for GCC flags. You can think of this as "enable feature". So -funsafe-math-operations should be read as (-f) (unsafe-math-operations). Not (-)(funsafe-math-operations).

[arcticbull]

I kind of like the idea the flag is sarcastically calling them very fun and very safe.

[dekhn]

don't forget libiberty which is linked in using -liberty (and freedom for all)

[gumby]

BTW that library was named by John Gilmore, who’s a pretty hardcore libertarian.

[cozzyd]

strangely I'm not aware of a libibre.

[dekhn]

oh, that's called -lmojito

[dahart]

Why do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it’s a 1 or maybe 2 LSB difference. At least in CUDA it is, and I think many (most?) people & situations can tolerate 22 bits of mantissa compared to 23, and many (most?) people/situations aren’t paying attention to inf/nan/exception issues at all.

I deal with a lot of floating point professionally day to day, and I use fast math all the time, since the tradeoff for higher performance and the relatively small loss of accuracy are acceptable. Maybe the biggest issue I run into is lack of denorms with CUDA fast-math, and it’s pretty rare for me to care about numbers smaller than 10^-38. Heck, I’d say I can tolerate 8 or 16 bits of mantissa most of the time, and fast-math floats are way more accurate than that. And we know a lot of neural network training these days can tolerate less than 8 bits of mantissa.

[jcranmer]

Here are some of the problems with fast-math:

* It links in an object file that enables denormal flushing globally, so that it affects all libraries linked into your application, even if said library explicitly doesn't want fast-math. This is seriously one of the most user-hostile things a compiler can do.

* The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code).

* Fast-math precludes you from being able to use NaN or infinities, and often even being able to defensively test for NaN or infinity. Sure, there are times where this is useful, but an option you might generally prefer to suggest for an uninformed programmer would rather be a "floating-point code can't overflow" option rather than "infinity doesn't exist and it's UB if it does exist".

* Fast-math can cause hard range guarantees to fail. Maybe you've got code that you can prove that, even with rounding error, the result will still be >= 0. With fast-math, the code might be adjusted so that the result is instead, say, -1e-10. And if you pass that to a function with a hard domain error at 0 (like sqrt), you now go from the result being 0 to the result being NaN. And see above about what happens when you get NaN.

Fast-math is a tradeoff, and if you're willing to except the tradeoff it offers, it's a fine option to use. But most programmers don't even know what the tradeoffs are, and the failure mode can be absolutely catastrophic. It's definitely an option that is in the "you must be this knowledgeable to use" camp.

[fl0ki]

> The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code).

This already shouldn't be assumed, because even the same code, compiler, and flags can produce different floating point results on different CPU targets. With the world increasingly split over x86_64 and aarch64, with more to come, it would be unwise to assume they produce the same exact numbers.

Often this comes down to acceptable implementation defined behavior, e.g. temporarily using an 80-bit floating register despite the result being coerced to 64 bits, or using an FMA instruction that loses less precision than separate multiply and add instructions.

Portable results should come from integers (even if used to simulate rationals and fixed point), not floats. I understand that's not easy with multiplayer games, but doing so with floats is simply impossible because of what is left as implementation-defined in our language standards.

[jcranmer]

This advice is out-of-date.

All CPU hardware nowadays conforms to IEEE 754 semantics for binary32 and binary64. (I think all the GPUs now have non-denormal-flushing modes, but my GPU knowledge is less deep). All compilers will have a floating-point mode that preserves IEEE 754 semantics assuming that FP exceptions are unobservable and rounding mode is the default, and this is usually the default (icc/icx is unusual in making fast-math the default).

Thus, you have portability of floating-point semantics, subject to caveats:

* The math library functions [1] are not the same between different implementations. If you want portability, you need to ensure that you're using the exact same math library on all platforms.

* NaN payloads are not consistent on different platforms, or necessarily within the same platform due to compiler optimizations. Note that not even IEEE 754 attempts to guarantee NaN payload stability.

* Long double is not the same type on different platforms. Don't use it. Seriously, don't.

* 32-bit x86 support for exact IEEE 754 equivalence is essentially a "known-WONTFIX" bug. (This is why the C standard implemented FLT_EVAL_METHOD). The x87 FPU evaluates everything in 80-bit precision, and while you can make this work for binary32 easily (double rounding isn't an issue), though with some performance cost (the solution involves reading/writing from memory after every operation), it's not so easy for binary64. However, the SSE registers do implement IEEE 754 exactly, and are present on every chip old enough to drink, so it's not really a problem anymore. There's a subsidiary issue that the x86-32 ABI requires floats be returned in x87 registers, which means you can't properly return an sNaN correctly, but sNaN and floating-point exceptions are firmly in the realm of nonportability anyways.

In short, if you don't need to care about 32-bit x86 support (or if you do care but can require SSE2 support), and you don't care about NaNs, and you bring your own libraries along, you can absolutely expect to have floating-point portability.

[1] It's actually not even all math library functions, just those that are like sin, pow, exp, etc., but specifically excluding things like sqrt. I'm still trying to come up with a good term to encompass these.

[zozbot234]

> It's actually not even all math library functions, just those that are like sin, pow, exp, etc., but specifically excluding things like sqrt. I'm still trying to come up with a good term to encompass these.

Transcendental functions. They're called that because computing an exactly rounded result might be unfeasible for some inputs. https://en.wikipedia.org/wiki/Table-maker%27s_dilemma So standards for numerical compute punt on the issue and allow for some error in the last digit.

[fl0ki]

Not sure if this is a spooky coincidence, but I happened to be reading the Rust 1.75.0 release notes today and fell into this 50-tab rabbit hole: https://github.com/rust-lang/rust/pull/113053/

[account42]

> All CPU hardware nowadays conforms to IEEE 754 semantics for binary32 and binary64.

Is this out of date?

https://developer.arm.com/documentation/den0018/a/NEON-Instr...

[AshamedCaptain]

> this is usually the default

No, it's not. gcc itself still defaults to fp-contract=fast. Or at least does in all versions I have ever tried.

[AshamedCaptain]

> Often this comes down to acceptable implementation defined behavior,

I believe this is "always" rather than often when it comes to the actual operations defined by the FP standard. gcc does play it fast and loose (as -ffast-math is not yet enabled by default, and FMA on the other hand is), but this is technically illegal and at least can be easily configured to be in standards-compliant mode.

I think the bigger problem comes from what is _not_ documented by the standard. E.g. transcendental functions. A program calling plain old sqrt(x) can find itself behaving differently _even between different stepping of the same core_, not to mention that there are well-known differences between AMD vs Intel. This is all using the same binary.

[mabster]

I'm surprised by this, regarding sqrt. The standard stipulates correct rounding for simple arithmetic, including sqrt ever since 754 1985.

Unless of course we are talking about the 80 bit format.

If that's not the case, would be interested to know where they differ.

Unfortunately for the transcendental function the accuracy still hasn't been pinned down, especially since that's still an ongoing research problem.

There's been some great strides in figuring out the worst cases for binary floating point up to doubles so hopefully an upcoming standard will stipulate 0.5 ULP for transcendentals. But decimal floating point still has a long way to go.

[dahart]

Thank you, great points. I’d have to agree that disabling denorms globally is pretty bad, even if (or maybe especially if) caring about denorms is rare.

> Fast-math can cause hard range guarantees to fail. Maybe you’ve got code that you can prove that, even with rounding error, the result will still be >= 0.

Floats do this too, it’s pretty routine to bump into epsilon out-of-range issues without fast-math. Most people don’t prove things about their rounding error, and if they do, it’s easy for them to account for 3 ULPs of fast-math error compared to 1/2 ULP for the more accurate operations. Like, nobody who knows what they’re doing will call sqrt() on a number that is fresh out of a multiplier and might be anywhere near zero without testing for zero explicitly, right? I’m sure someone has done it, but I’ve never seen it, and it ranks high on the list of bad ideas even if you steer completely clear of fast-math, no?

I guess I just wanted to resist the unspecific parts of the FUD just a little bit. I like your list a lot because it’s specific. Fast-math does carry some additional risks for accuracy sensitive code, and clearly as you and others showed, can infect and impact your whole app, and it can sometimes lead to situations where things break that wouldn’t have happened otherwise. But I think in the grand scheme these situations are quite rare compared to how often people mess up regular floating point math. For a very wide swath of people doing casual arithmetic, fast-math is not likely to cause more problems than floats cause, but it’s fair to want to be careful and pay attention.

[PaulDavisThe1st]

> I’d have to agree that disabling denorms globally is pretty bad,

and yet, for audio processing, this is an option that most DAWs either implement silently, or offer users the choice, because denormals are inevitable in reverb tails and on most Intel processors they slow things by orders of magnitude.

[dahart]

I would think for audio, there’s no audible difference between a denorm and a flushed zero. Are there cases where denorms are important to audio?

[mabster]

For game development we had them off as well because of performance issues. Most stuff calculates around 0 so it was pretty common to trigger denorms.

The slowing down on Intel platforms has always frustrated me because denorms provide nice smoothing around 0.

At the same time it was nice only having to consider normal floating point when trying to get more accuracy out of calculations, etc.

[mort96]

The scary thing IMO is: your code might be fine with unsafe math optimisations, but maybe you're using a library which is written to do operations in a certain order to minimise numerical error, and unsafe math operations changes the code which are mathematically equivalent but which results in many orders of magnitude more numerical error. It's probably fine most of the time, but it's kinda scary.

[dahart]

It shouldn’t be scary. Any library that is sensitive to order of operations will hopefully have a big fat warning on it. And it can be compiled separately with fast-math disabled. I don’t know of any such libraries off the top of my head, and it’s quite rare to find situations that result in orders of magnitude more error, though I grant you it can happen, and it can be contrived pretty easily.

[planede]

You can't fully disable fast-math per-library, moreover a library compiled with fast-math might also introduce inaccuracies in a seemingly unrelated library or application code in the same executable. The reason is that fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways.

[dahart]

> You can’t fully disable fast-math per library

Can you elaborate? What fast-math can sneak into a library that disabled fast-math at compile time?

> fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways.

I wasn’t aware of this, I would love to see some documentation discussing exactly what happens, can you send a link?

[londons_explore]

you're gonna hive to give us a concrete real world example to convince most of us...

[mort96]

I don't typically thoroughly read through the documentation for all the dependencies which my dependencies are using.

But you're correct that it's probably usually fine in practice.

[dahart]

That’s fair. Ideally transitive dependencies should be completely hidden from you. Hopefully the author of the library you include directly has heeded the instructions of libraries they depend on.

Hey I grant and acknowledge that using fast-math carries a little risk of surprises, we don’t necessarily need to try to think of corner cases. I’m mostly pushing back a little because using floats at all carries almost as much risk. A lot of people seem to use floats without knowing how inaccurate floats are, and a lot of people aren’t doing precision analysis or handling the exceptional cases… and don’t really need to.

[light_hue_1]

Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong!

People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums to 0, it can cause catastrophic cancellation, etc.

Please stop giving people terrible advice on a topic you're totally unfamiliar with.

[alexey-salmin]

> It can reorder your computations so that something that used to sum to 1 now sums to 0, it can cause catastrophic cancellation, etc.

Yes, and it could very well be that the correct answer is actually 0 and not 1.

Unless you write your code to explicitly account for fp associativity effects, in which case you don't need generic forum advice about fast-math.

[dahart]

I only do numeric computation, I don’t work in machine learning. Sorry your assumptions are incorrect, maybe it’s best not to assume or attack. I didn’t exactly advise using fast math either, I asked for reasoning and pointed out that most casual uses of float aren’t highly sensitive to precision.

It’s easy to have wrong sums and catastrophic cancellation without fast math, and it’s relatively rare for fast math to cause those issues when an underlying issue didn’t already exist.

I’ve been working in some code that does a couple of quadratic solves and has high order intermediate terms, and I’ve tried using Kahan’s algorithm repeatedly to improve the precision of the discriminants, but it has never helped at all. On the other hand I’ve used a few other tricks that improve the precision enough that the fast math version is higher precision than the naive one without fast math. I get to have my cake and eat it too.

Fast math is a tradeoff. Of course it’s a good idea to know what it does and what the risks of using it are, but at least in terms of the accuracy of fast math in CUDA, it’s not an opinion whether the accuracy is relatively close to slow math, it’s reasonably well documented. You can see for yourself that most fast math ops are in the single digit ulps of rounding error. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....

[thechao]

+1. I'm years away from fp-analysis, but do the transcendental expansions even converge in the presence of fast-math? No `sin()`, no `cos()`, no `exp()`, ...

[dahart]

Well there are library implementations of fast-math trancendentals that offer bounded error, and a million different fast sine approximation algorithms, so, yes? This is why you shouldn’t listen to FUD. The corner cases are indeed frustrating for a few people, but most never hit them, and the world doesn’t suddenly break when fast math is enabled. I am paid to do some FP analysis, btw.

[usefulcat]

> Fast math is only slightly less accurate

'slightly'? Last I checked, -Ofast completely breaks std::isnan and std::isinf--they always return false.

[dahart]

Hopefully it was clear from the rest of my comment that I was talking about in-range floats there. I wouldn’t necessarily call inf & nan handling an accuracy issue, that’s more about exceptional cases, but to your point I would have to agree that losing std::isinf is kinda bad since divide by zero is probably near the very top of the list of things most people using floats casually might have to deal with.

Which compiler are you using where std::isinf breaks? Hopefully it was also clear that my experience leans toward CUDA, and I think the inf & nan support works there in the presence of NVCC’s fast-math.

[usefulcat]

My experience is with gcc and clang on x86. I generally agree with you regarding accuracy, which is why I was quite surprised when I first discovered that -Ofast breaks isnan/isinf.

Even if I don't care about the accuracy differences, I still need a way to check for invalid input data. The upshot is that I had to roll my own isnan and isinf to be able to use -Ofast (because it's actually the underlying __builtin_xxx intrinsics that are broken), which still seems wrong to me.

[xigoi]

They are talking about -ffast-math, not -Ofast.

[gumby]

From the gcc manual:

-Ofast

Disregard strict standards compliance. -Ofast enables all -O3 optimizations. It also enables optimizations that are not valid for all standard-compliant programs. It turns on -ffast-math, -fallow-store-data-races and the Fortran-specific -fstack-arrays, unless -fmax-stack-var-size is specified, and -fno-protect-parens. It turns off -fsemantic-interposition.

[alexey-salmin]

If your code ventures into the domain where fast-math matters and you're not a mathematician trying to solve a lyapunov-unstable problem with very tricky numeric methods, then most likely your code is already broken.

[planede]

Another PSA is that dynamic libraries compiled with fast-math will also introduce inaccuracies in unrelated libraries in the same executable, as they introduce dynamic initialization that globally changes the floating point environment.

[pavlov]

This would only affect code that uses the old-school x87 floating point instructions, though? The x87 FPU unit indeed has scary global state that can make your doubles behave like floats in secret and silence.

I would think practically all modern FPU code on x86-64 would be using the SIMD registers which have explicit widths.

[borodi]

So it was a bit more pervasive than this, the issue was that flushing subnormals (values very close to 0) to 0 is a register that gets set, so if a library is built with the fastmath flags and it gets loaded, it sets the register, causing the whole process to flush it's subnormals. i.e https://github.com/llvm/llvm-project/issues/57589

[jcranmer]

> This would only affect code that uses the old-school x87 floating point instructions, though?

Actually, no, the x87 FPU instructions are the only ones that won't be affected.

It sets the FTZ/DAZ bits, which exist for SSE instructions but not x87 instructions.

[mhh__]

You're mistaking something else for the rounding mode and subnormal handling flags.

[cogman10]

Ehh, not so much inaccurate, more of a "floating point numbers are tricky, let's act like they aren't".

Compilers are pretty skittish about changing the order of floating point operations (for good reason) and ffast-math is the thing that lets them transform equations to try and generate faster code.

IE, instead of doing "n / 10" doing "n * 0.1". The issue, of course, being that things like 0.1 can't be perfectly represented with floats but 100 / 10 can be. So now you've introduced a tiny bit of error where it might not have existed.

[rwmj]

It isn't just that. -ffast-math also allows the compiler to ignore infinites. In fact for GCC with -ffast-math, isinf always returns false. Something similar happens for NaNs/isnan.

[cogman10]

I lump this into "floating points are tricky". NaNs and inf are definitely legitimate floating point values. They are also things that a lot of applications will break on they ever encounter them.

[phkahler]

I've never understood why generating exceptions is preferable to just using higher precision.

[gumby]

Higher precision isn’t always available. IEEE 754 is an unusually well-thought-through standard (thanks to some smart people with a lot of painful experience) and is pretty good at justifying its decisions, some of which are surprising (far from obvious) to anyone not steeped in it.

[cogman10]

The main problem with floats in general is they are designed primarily for scientific computing.

We are fortunately starting to see newer (well, not that new now) CPU instructions like FMA that make more accurate decimal representations not take such huge performance hits.

[adgjlsfhk1]

what does fma have to do with decimal numbers?

[dahart]

On a GPU, higher precision can cost between 2 and 64 times more than single precision, with typical ratios for consumer cards being 16 or 32. Even on the CPU, fp64 workloads tend to run at half the speed on real data due to the extra bandwidth needed for higher precision.

[mhh__]

One of the things that you can do with D and as far as I know Julia is enable specific optimizations locally e.g. allow FMAs here and there, not globally.

fast-math is one of the dumbest things we have as an industry IMO.

[ChrisRackauckas]

Totally agreed. In Julia we use https://github.com/SciML/MuladdMacro.jl all over the place so that way it's contextual and does not bleed into other functions. fast-math changing everything is just... dangerous.