[bcantrill]

At Joyent, we deployed public-facing multitenant workloads based on zones (and before that, jails) for many years. We seriously trusted it -- and had serious customers who seriously depended on it. So, now you know someone!

[psanford]

To be fair, y'all had some serious vulnerabilities, including zone escapes and arbitrary kernel memory reads, discovered by @benmmurphy.

[bcantrill]

Yes, though I would like to believe that Ben's responsible disclosure coupled with our addressing those vulns (and auditing ourselves for similar) reflect exactly that seriousness around multitenant security. And for whatever it's worth, one of those vulnerabilities -- which was a bug in my code! -- very much informed by own thinking about the inherent unsafety of C, underscoring the appeal of Rust. So I am grateful in several dimensions!

[tptacek]

If you have a kernel implemented in Rust, (1) you should shout that from the rooftops and (2) use whatever isolation mechanism you like on it.

[bsder]

They're starting with the bootloader and management engine. That's a tough enough ocean to boil.

Give them some time to get Rust above that.

[pjmlp]

Sadly Apple decided for a safe dialect of C for similar purposes e.g. iBoot, where they could have gone with Swift or Rust instead.

Very big ocean indeed.

[tptacek]

To this, all I can say is that I spent from 2005-2014, and then from 2016-2020, doing nothing but security evaluations of products, probably about 60% of which were serverside multitenant SAAS systems of one form or another, and I don't remember ever evaluating (or overseeing the evaluation of) a system that relied on Jails or Zones. Lots of Docker! And, until a few years ago, multitenant Docker isolation was an infamous joke! I'm not sticking up for it!

You can look at the recent history of Linux kernel LPEs --- there has been sort of a renaissance because of mobile devices --- and count all the ways any shared-kernel multitenant system would have broken down. At the end of the day, it's not so much about predicting whether your system can get owned up (it can), so much as: "what do I need to do when there is a kernel LPE announced on my platform". If you're doing shared-kernel isolation, the right answer to that question is usually "fire drill". It's not a noodley thought-leadership kind of question; it's a simple, practical concern.

[wmf]

There were also tons of providers who trusted Linux containers for VPS hosting.

[tptacek]

How'd that turn out?

[wmf]

I haven't heard any stories of people being hacked via container escape, but the whole VPS industry was so low-stakes that maybe customers didn't expect good isolation anyway.

[unixhero]

And needless to say it became a billion dollar business, with a great product.

[psanford]

They were acquired for $170m.

[unixhero]

I stand corrected. Still great product, business and team.

[tptacek]

I'm sure they're great. No part of what I have to say about this has anything to do with how competent they are.

[mixmastamyk]

Security requirements (and awareness) have increased over the years, have they not?

[bcantrill]

They definitely have! And we had a (zones-based) public cloud through it all. On that note, Alex Wilson's description of working with Robert Mustacchi on mitigating Meltdown by adding KPTI to illumos[0] definitely merits a read!

[0] https://blog.cooperi.net/a-long-two-months