I'm doing some research that shows this isn't the case. You can have a system where enabling the debugger has zero impact on runtime performance, via dynamic deoptimization.
http://www.lifl.fr/dyla14/papers/dyla14-3-Debugging_at_Full_...