[j4_james]

One technique I remember being used in DOS apps from many years ago was that the code would be encrypted in such a way that the next instruction to be executed would only be decrypted immediately before it was run. This was achieved by setting up the single step interrupt as the decrypter, and running the code in single step mode.

The fact that the code was encrypted meant the debugger couldn't disassemble it in any meaningful way, and also made it impossible to set a breakpoint (since the breakpoint would just end up being "decrypted" into some other opcode that would inevitably crash). The debugger also couldn't step through the code, because taking over the single step interrupt would prevent the decrypter from running, so you'd just be stepping through garbage.

The way I worked around this was by writing a debugger that could hook the single step interrupt in such a way that it still forwarded the interrupt onto the previous hook. I still couldn't set breakpoints, but I could step through the code, watching it decode itself as it proceeded.

[caf]

If you're single-stepping you don't need to patch the instruction stream for a breakpoint; you can just test the instruction pointer against the break address at each step.

[Annatar]

Not every processor has the means to perform direct comparisons against the program counter; as far as I know 6502 and MC68000 families can not do direct comparisons with the program counter.

[caf]

It's the program counter of the state that was interrupted by the single-step trap that's of interest, and that program counter is generally pushed onto the stack by the trap.

For example, this is how the trace exception on the M68k works - the program counter of the next instruction to be executed can be read off the stack by the exception handler. The 6502 doesn't have built-in software single-stepping but the same effect was sometimes achieved by tying a short timer to the NMI - and when the NMI is asserted, the interrupted program counter is pushed onto the stack.