I would guess the processor would just cause an exception/interrupt and it would just call an OS level exception/interrupt handler which would probably tell the user what the exception/interrupt was that occured; in this case an unsupported instruction.
[0]: https://github.com/xoreaxeaxeax/sandsifter