[lmm]

> Stopping at one 3ms optimization is not going to move mountains. But doing that 10 times is already 30ms. Imagine that you found 100 micro optimizations. Now we are talking!

300ms still doesn't put a dent in 8 or 9 seconds.

> So keep calm, optimize on. This is how compilers get better over time.

So work with the compiler rather than against it. As others have pointed out, with the correct -march setting you get code that's just as good as this hand-tuned assembly - and much more maintainable, and the correct compiler setting will improve the rest of your code as well.