[menaerus]

And do you know about Linux? After cursory search I couldn't find anything as well. In this case it seems very expensive, if not impossible without the huge penalty, to implement such a workaround in the hardware.

Heterogeneous compute aware kernel scheduler seems like the most natural approach to me.

[murderfs]

The kernel can't do anything here, unless your answer is "threads can't ever migrate between types of cores, so cross your fingers on which core you spawn on". The kernel doesn't know what flavor of ISA userspace is expecting, because userspace doesn't know what flavor it wants until it actually gets to run: symbol multiversioning lets you compile a function for multiple versions of an instruction set and pick the best one at runtime.

You could imagine the kernel being clever and handling the first SIGILL in hardware and then marking the process as big-core-only, but you can't handle this generally since you can have new instructions get encoded as nop (I believe the AArch64 PAC instructions do that?). You could also have an ifunc resolver intentionally call the instruction and expect and handle the signal.

The solution is to make your hardware actually work, and not claim to support instructions that do not actually exist on all of your cores.