[rwmj]

I think what Intel are doing with Clear Containers is really interesting. They are encapsulating containers inside VMs, avoiding the security problems of containers.

To do this efficiently they've had to make a bunch of changes on the VM side so the overhead is much smaller than an ordinary VM (of the order of 150ms and 20MB of RAM). I've also been looking at this and am hoping to give a talk about it at the KVM Forum in August (http://events.linuxfoundation.org/events/kvm-forum).

[colemickens]

Windows Containers with Hyper-V isolation are similar. https://msdn.microsoft.com/en-us/virtualization/windowsconta...

[sillysaurus3]

150ms for which operation?

[philips]

150ms to boot the VM that is running the container processes with the stripped down Linux Kernel, lkvm, and DAX. This is about what we observe in the rkt Clear Containers "stage1": https://coreos.com/blog/rkt-0.8-with-new-vm-support/

One note: you can run multiple "container processes", like redis and a redis dashboard, inside of these Clear Container VMs. This means in the case of Kubernetes we will only incur the cost of the startup time and Kernel/init overhead once per pod instead of once per process.

[sillysaurus3]

This is going to sound amusing at best, but would you clarify what it means to boot?

To justify the question a bit: booting traditionally meant physically turning a system on. The boot time included BIOS initialization, a concept now blurred by the advent of virtualization.

150ms is such an absurdly short amount of time that I'm left wondering what booting is in this context.

[wmf]

VMs usually do have BIOS (sometimes you can see it flicker on the screen) but like NeutronBoy said, the hypervisor just creates the virtual hardware devices in a pre-initialized state so the BIOS has to do almost no work and it completes in a fraction of a second. Clear Containers boots even faster by not using BIOS; the hypervisor directly loads the kernel and initrd into RAM. So in this case "booting" means starting the kernel, mounting the root filesystem (accelerated using DAX), running init, starting dockerd, etc.

[rwmj]

Actually Clear Containers does have a BIOS, because Linux requires one eg to read E820 data and to set up the virtual video. However the one Clear Containers uses (from kvmtool) is extremely minimal -- it's literally enough of a BIOS just to answer the int calls that modern Linux makes at boot and that's all. IIRC it's hundreds of lines of code only.

[mikewhy]

> 150ms is such an absurdly short amount of time that I'm left wondering what booting is in this context.

Clear Linux was announced about a year ago, and it does boot absurdly quickly

https://clearlinux.org

https://lwn.net/Articles/644675/

[philips]

It does this thanks to a technology called DAX and the fact that systemd boots really fast.

https://www.kernel.org/doc/Documentation/filesystems/dax.txt

[rwmj]

DAX is a small part of it, but Intel made many changes throughout the stack, mostly to the Linux kernel.

[NeutronBoy]

> This is going to sound amusing at best, but would you mind defining what it means to boot?

I imagine that most of the time would be in mocking some/all of the hardware interfaces to present to the VM, and running your init processes (and all that entails for whatever OS you're running).

[e12e]

For perspective, I just timed (not very well) how long it takes windows to run the c-program "exit": "int main() { return 0; }", compiled with gcc 4.8.1, -O3 -std=c11 -Wall, stripped[1]. From a warm disc cache it takes ~3ms. From cold(er) it takes ~19s.

Taking a 50x hit to run "exit" from a container doesn't sound bad, but it doesn't sound all that far fetched either.

[1] time util from pstools, as installed by scoop.sh - similar to why gcc (not eg msvc - it's all in my path atm, no work needed :)

[e12e]

~19ms, obviously, not 19 seconds.

[geggam]

Because running a process in a VM doesnt meet what requirement ?

[rwmj]

Your question is confusing, what does it mean?

If you meant "running a process in a VM meets what requirement?", the requirement is security, which as the paper here proves is not available with simple containers running as processes on the host.