[oconnore]

You can run https://gvisor.dev/ without any virtualization requirement. We use this to host user-submitted configurations (not arbitrary code, but arbitrary input to ~mostly trusted code).

Does this not meet your requirements?

[paulfurtado]

gvisor is awesome and works for particularly untrusted applications, but it's not a performance hit we'd be willing to take across the board and effectively only protects you from security bugs rather than other kernel issues. We run thousands of production database workloads, hundreds of load balancers, thousands web apps, ML jobs, batch processing, etc in kubernetes, most of which require as much performance as possible.

When an EBS volume for a pod goes impaired, if it's using xfs you can basically count the whole server as dead no matter how many xfs + block io timeouts you set. xfs will stop being able to mount/unmount any other filesystems once hung in an unmount call for one. With a proper VM, you'd passthrough the nvme device with pcie passthrough and the host would be totally unimpacted.

Also, gvisor's better mode requires kvm, but it's cool that it effectively functions with ptrace when you can't use kvm.