[justnoise]

The company I work at started with the idea that we would build an orchestration system for Unikernels. Very much in the same space as the NanoVMs folks. I totally applaud their work. We took a different path once our small team worked with unikernels for a bit (mostly OSv). During the early days we noticed that the unikernel landscape had technical shortcomings that would take an incredible amount of engineering effort to overcome and we found convincing users to trade Linux for a (mostly but not totally) compatible unikernel based system was an insurmountable hurdle. It was a fun experiment but, after timeboxing our work and taking stock of the landscape, we fell back to one of our original sentiments: A stripped down Linux is actually a pretty good substitute for a Unikernel for most applications (emphisis on "most).

We ended up pivoting to running a lightweight linux, based on Alpine and orchestrating everything using Kubernetes and Virtual Kubelet [1]. Shameless Plug! Pods are isolated on their own virtual machines that are booted for the pod, the underlying OS is rock solid and gives users all the great tools, bells and whistles you'd expect from a linux based system. Fewer surprises, easier development. We actually open sourced the system today.

[1] https://github.com/elotl/kip

[wwarner]

Great intel from on the ground. The story arc makes total sense. I also have starry eyes for orchestrated unikernels, but otoh linux is so configurable scaling down the kernel also seems very reasonable. How do the boot times compare? To me, the unikernel's chief appeal is that it could potentially start up in about the same amount of time it takes to load a native binary. Thanks for sharing.

[justnoise]

On bare-metal boot times are drastically different. OSv was sub-second while our Alpine images were 3-5 seconds depending on what services we needed. However, we were focused on running our system on cloud VMs, not bare-metal. In the cloud you can't get a VM to start booting in less than 20-30s so that order of magnitude difference turned into, at most a 10% difference in boot times.

In 2017 we measured the restart time in of our unikernel images in AWS to be 22 seconds, all that time was waiting for Zen (2017... no KVM yet) to get around to getting to the place where we could run our image. So for our use case, the boot time didn't actually matter, it was far overshadowed by everything else happening under the hood.

I should say: Unikernels do have their advantages and should be used in areas that can exploit those advantages: Fast boot, low attack surface, way better performance for some workloads. We had trouble finding the specialized customers in the cloud that needed any of a unikernel's positives so badly that they would take on a unikernel's shortcomings.