Could this be implemented at the OS level, i.e. whenever a proces launches, the OS generates a key that it will keep to itself and use to transparently encrypt all memory allocated by that process?
My first thought was to try to use 'containers' (cgroups) combined with the AMD secure memory extensions to achieve this type of isolation using as much off the shelf hardware as possible.
From the quick description it sounds like this provides a way of encrypting, per memory page, based on a symmetric key that is backed by some level of hardware encryption. It was not clear (in a quick read) how or where to specify the key by which an individual page is encrypted. That would be a critical component of comprehension with respect to identifying if this could be used to encipher individual processes and further isolate memory. It sounds like it might be possible to establish per-process memory isolation, which is probably the best level of security possible without resorting to entirely isolated hardware.
Per-process keys aren't really possible because memory can change process ownership (vmsplice) or be shared across processes (fork, page cache, memfd). It might be possible for pages marked MADV_DONTFORK
Additionally a per-process key does not help against spectre style attacks where you would trick the process into speculating on protected memory.
You'd probably want a hardware module to do that lest performance plummets. Memory controllers can already deal with ECC efficiently, adding a simple cypher on top of it should definitely be feasible.
Possibly, but memory is accessed using plain CPU instructions, so it would be hard to transparently encrypt all memory for an application at the kernel level. You do have virtual memory, but I dont think that could be leveraged for this. But who knows whats possible there, maybe if you align and address each memory value at the page boundaries and always force a page fault you could have a really poor implementation :)
Transparent disk encryption, not a problem since devices have filesystems which can implement encryption at that layer.
Modern Intel chips can encrypt memory on the fly without performance loss (SGX does this). However I think it's not exposed for non-enclave use. Perhaps it should be.
Note: inside the enclave there is a performance loss but that's due to MAC checks. If you just want encryption without integrity against tampering you don't need that.
Yes. Most research makes CPU modifications since that makes the most sense. Sometimes they try to use OS-level techniques. Here's a survey showing some of each:
https://en.wikichip.org/wiki/x86/sme
https://www.kernel.org/doc/Documentation/x86/amd-memory-encr...
From the quick description it sounds like this provides a way of encrypting, per memory page, based on a symmetric key that is backed by some level of hardware encryption. It was not clear (in a quick read) how or where to specify the key by which an individual page is encrypted. That would be a critical component of comprehension with respect to identifying if this could be used to encipher individual processes and further isolate memory. It sounds like it might be possible to establish per-process memory isolation, which is probably the best level of security possible without resorting to entirely isolated hardware.