[superbatfish]

If you are willing to accept the various caveats that come with -ffast-math for your own library, then it looks like it's okay to use -ffast-math during compilation, but not during linking (because it links in crtfastmath.so, as the author points out).

Your own compiled library functions will use the optimizations, but won't force the weird FPU register modes on the rest of the process.

[account42]

Pay close attention to your build system though. Many will pass all compiler flags to the linker driver as that is needed to make sure that they affect LTO.