[tryp]

An invalid or debug instruction causes the processor to throw an exception and jump to a specific address where "handler" code should exist. This is managed by the OS, so in this case, the OS would look up the proper thing to do as configured by the earlier ptrace calls -- probably executing your debugger code to let you inspect variables and memory. When you're done inspecting the proper instruction gets re-installed in the target process, the OS restores state saved by the exception trap, and transfers execution back to the debugged process. Most processors have a single-step flag so that the next instruction to execute and the following trigger the exception.