I think one of the exceptions is heavily optimized code, though the surrounding code can still be maintainable and clear, with the weird stuff to tickle the compiler or inline assembly being heavily commented.
For those kinds of cases, I love having as-simple-as-possible reference code checked in alongside it, even if it's just #ifdef'd out for common builds. Having a baseline makes it sooo much easier to understand and debug the optimized code.
Ideally, the slow code is available alongside the fast code and the unit tests confirm they are sufficiently equivalent. If that's not possible, the go-fast bits need to be refactored until it is possible. Don't forget that error handling is part of the equivalency testing.
Beyond unit tests, this seems like a great opportunity to leverage property-based testing, at least for pure functions; checking that both versions return the same results is an easy property to think of.