[billypilgrim]

Also: malware often tries to detect a VM or an emulator too, for example Windows Defender uses an emulator internally to detonate samples, and there are attempts by malware to detect this and change the behavior to something benign.

[EvanAnderson]

Way back in the early 90s Thunderbyte Antivirus' TBCLEAN would use the x86 trap flag to single step viruses up to the point where they restored the original entrypoint of the infected program, then write the "cleaned" program back to disk. They used the CPU single-step as a hack to alleviate needing to write an emulator.

The virus writer Priest figured out he could detect being run under single-stepping, and manipulate the stack and trap flag to re-vector control from TBSCAN to a destructive routine that trash the user's hard disk (but otherwise just run normally when not in the presence of TBCLEAN).

He later used this idea as the basis for the "emulating tracer" (in Natas, for sure-- but I think present in some earlier code too-- I don't remember what, thought) using single-step interrupt calls to trace thru resident antivirus programs to find original BIOS and DOS interrupt vectors and "call past" them (to prevent detection and do stealth).

His tracer decoded the next instruction to detect every method by which the trap flag state could be leaked to or mutated bu the traced code. He would emulate and step over any of these "privileged" instructions", presenting a sanitized state to the code under trace. It wasn't a full x86 emulator and could not have handled code that used trap-based anti-debug. That would have required a full emulator (and that way lies madness).

When VMware virtualized x86 I thought about Priest's code. Defender and other AV running samples under emulation makes me think about it too. So does this article.