This button was added to specifically combat that, and even then it means you can't use breakpoints at all. IMO the missing link is a way to specifically ignore a single debugger call.
The debugger lets you edit the code in the VM, so if there's a debugger call you don't want you can simply delete it and continue and it'll not show up again until you refresh