Actually, my preferred solution would be to eliminate the notion of distributing machine code binaries entirely, but that's a bit beyond the scope of these discussions. ;-)
No, creating a block of machine code bytes to execute would be a privileged operation. All code would run through a privileged CPU-specific compiler first, and there'd be no way to run raw machine code bytes otherwise.
If there are bugs that can be exposed through various machine code patterns, the compiler can centralize the restrictions of what may be executed, enforce runtime checks, or prevent certain instructions from being used at all. Security or optimization updates would affect all running programs automatically. Granted, these current speculative vulnerabilities would be much more difficult to statically detect.
But it would follow the crazy gentoo dream of having everything optimized for your environment better, allow much better compatibility across systems, and prevent entire classes of privilege escalation issues.
> no way to run raw machine code bytes otherwise [...] restrictions of what may be executed, enforce runtime checks, or prevent certain instructions from being used at all [...] everything optimized for your environment better, allow much better compatibility across systems and prevent entire classes of privilege escalation issues.
So... basically re-inventing Java? :)
"Raw machine code bytes" aren't distributed but occur through the privileged JVM and its just-in-time compiler, the byte-code verifier enforces restrictions on what data-access patterns and where instructions can be used, the JVM for a particular OS has optimizations for that environment, and sandboxing (while imperfect) blocks some classes of privilege escalation issues.
Don't get me wrong, I'm not saying Java is perfect or that the underlying goal isn't good, I'm just happily amused by this sense of "everything old is new again."
Well, to me Java is still new tech. ;-) But yes, it's certainly a reasonable sampling into non-machine code distribution, and enforcement of security rules when actually running/JITting the code, as were some mainframe developments before then.
Of course, Java certainly does have some higher level weaknesses as in the introspection API kerfuffle a while back, and is too locked into its Object Obsessed design for it to be a truly general purpose object code format.
I've been thinking along the same lines for the last few years. If you did this, you could have a multi-user operating system in a single address space and avoid the cost of interrupts for system calls (which would just be like any other function call).
We'd need a better binary representation of uncompiled code, then. Moving around lots of code as ascii is kind of suboptimal... I wouldn't want that. By all means, show it as text to the user, but don't store it that way.
and what if I wrote a compiler that doesn't heed any of your security concerns? It would still compile to machine code and continue to be able to exploit things Spectre/Meltdown style? Or am I off here?
You'd only be able to run it on your system. At least, without other means of breaching the low level secured configuration of someone else's machine, because that's where the One True Compiler for that system lives.
cool .. I think I get it. It's like compiler/instruction based DRM ... CPU specific permission to run code. Maybe they can leverage existing TPM chips to do this...
I just don't want to see performance being decimated as a trade off for security, if at all possible.