[mdriley]

When you say "control over the ISA", I'll assume you mean "precise control over the emitted instructions".

In which case: yes! That's Speculative Load Hardening (https://llvm.org/docs/SpeculativeLoadHardening.html). SLH tries to squash side-channels by preventing any speculatively-loaded data from being forwarded to dependent instructions until proving that branch prediction followed the right path.

But this undoes a lot of the performance that microarchitectures have added through branch prediction, since dependent memory loads (think linked list entries, or C++ vtables) are stalled behind full resolution of the branch condition.

If you're doing nontrivial compute, you can end up ahead performance-wise by splitting the computation into a separate process and invoking it via IPC. Now you don't need SLH because the untrusted process doesn't have long-lived secrets in its address space.

[reitzensteinm]

No, I mean specifically if you're building your own CPUs and can add instructions. You add the ability to set a hardware mask that all values are passed through before they're used as addresses for loads and stores including speculation. Loads and stores that fall outside the masked region will simply wrap around.

In your JIT, you enter and leave this mode before and after running user code to ensure it can't escape its region.

This would be a lot of work to pull off and would require custom hardware and software, but (at least as far as I can tell) it should work.

[gpderetta]

This is basically segments. It would work as long as your implementation doesn't have Meltdown-like vulnerabilities, i.e. speculation past hardware enforcend protection. We know it can be done because there are high performance CPUs which are not vulnerable to Meltdown.