[eatonphil]

The thing that kills me about all of this is how hard it is to do it right. I wish there were a dumbed down version of containers and orchestrators for people trying to do basic multi-tenant compute in a SaaS and don't care a ton about the best performance.

Would I be generally ok if I use gvisor to give a shell environment to customers and just keep the host up to date?

Or is using containers just relatively pointless for multitenant compute in a SaaS compared to giving customers virtual machines?

If you can't imagine the kind of SaaS I'm talking about, think something along the lines of Github's new online IDE, CodeSpaces.

[nonameiguess]

Multitenancy is difficult with containerization and not something I would recommend. It isn't what the technology is intended for. The ultimate example of multitenancy is actual platform and infrastructure providers and they all do it by giving you VMs because type I hypervisors are actually designed to do this kind of thing. Breakouts are always still possible when two processes are on the same physical server, but it's never as trivial as figuring out how to mount the kernel virtual filesystems.

I say this as a Kubernetes consultant. If you want "multitenancy" in the sense of distinct product or application teams all employed by the same parent company or organization, it's fine. But if you're talking truly different organizations with no implied trust between them, don't put them on a shared cluster.

I'm kind of curious how Github does this, because you can still get very minimalistic with VMs. Make the startup script for your application something that also mounts the filesystems it needs and name it /sbin/init and you just made yourself a poor man's unikernel.

[cpuguy83]

I'll be devil's advocate and say breakouts are totally possible with VM's, just by different vectors.

The vast majority of container breakouts are due to bugs in the control plane and not so much the kernel. The same is likely true for VMM's/hypervisors until those really started getting mature.

dotCloud and and Heroku are both examples of multi-tenant containers.

[raesene9]

That's very true, although I think there's a difference in attack surface size between the three isolation options (process based, sandbox based, hypervisor based).

I think the challenge for process isolation container based stacks (as I'm sure you know :) ) is that there's multiple components/groups involved in security and then there's co-ordination with the underlying Linux kernel as well, which makes things tricky, as Linux kernel devs will have potentially differing goals to the container people (e.g. the challenges about how to handle the interaction of new syscalls and seccomp filters)

If you compare that to something like gVisor, where there's essentially a single group responsible for creating/maintaining the sandbox, it's an easier task for them.

[paxys]

I think "dumbed down" and "multi-tenant compute" aren't compatible. No company needs to do multi-tenant compute by default. If you do, you are in the cloud hosting/infrastructure business (whether you like it or not) and should be expected to have the knowledge necessary to run such an operation.

[eatonphil]

While that's a common sentiment in some topics in tech; I think the general intent, if not actual result, of progress in tech is to make things faster, more secure and easier.

So forgive me for asking. :)

[raesene9]

Multi-Tenant Kubernetes is straight up difficult to do well, especially where you're talking hard multi-tenancy for external customers.

There was a good report that covered a lot of the risks and mitigations here https://raw.githubusercontent.com/salesforce/kubernetes-cont...

But even then that had limited scope and didn't cover things like networking.

[mac-chaffee]

There's a set of benchmarks for multi-tenant kubernetes clusters that might prove useful (although they could use more depth): https://github.com/kubernetes-sigs/multi-tenancy/tree/master...

[simonebrunozzi]

Reinventing the wheel, all the time. Early VMware (with VMs) had a much better sense of product than Google had with K8S.