99% of your code doesn't need optimization, period. If it's not inside at least two loops it almost certainly doesn't meaningfully contribute to the total runtime.
That's only true for application code. If you write libraries, you can't know how your library functions will be called and have to assume that most of them could end up in performance-critical code paths.
That case is easy to hit with compilers or annotation and such features. It is still one thing to optimize but impact is typically big.