[druiid]

Is it me or is Ubuntu kind of like the Sony of server software? It always seems like they are developing concurrent solutions to fit into their model of doing things.

Where Sony pushes their own special formats like Memory Stick, Ubuntu pushes Upstart, Juju and now LXD. I think in the end this isn't entirely helpful to the ecosystem as a whole when you have Ubuntu attempting to push their special formats of things while not bringing all that much more to the table. The question of how systemd compares to upstart might be something to consider with my upstart comparison, but essentially all I see from Ubuntu is pushing their brand of tooling and most of it is very Ubuntu specific.

On the other hand RedHat usually releases software that can (and usually does) make it to other distributions and ecosystems. I'm not entirely sure what LXD is going to bring to the table beyond what Docker or similar utilities offer, and like many Ubuntu projects it currently seems to be VERY light on documentation. Actually, where do I even access the documentation for this project? Oversights like this are what killed Juju or MAAS for me and yet Ubuntu pushes those projects like crazy at every conference I've seen them at (Gophercon for instance).

[vezzy-fnord]

Upstart seems like an odd example to give, given that it was the first sysvinit alternative that actually made it into major distributions. RHEL 6 (thus CentOS 6), ChromeOS and some of the earlier Fedora versions make use of it. This was after an attempt to port launchd failed, because of licensing issues. systemd came several years later.

Red Hat is a different business model. Their venturing into the cloud is more recent. Historically, they've been more into the support business, and this necessitated having a lot of people fix bugs in the Linux ecosystem. That and their acquisition of Cygnus Solutions means they're the de facto gatekeepers of the Linux kernel and much of userspace.

Canonical is a more Apple-like company. They care about being internally consistent and formulating their own brand, interacting with the outside only where necessary.

[druiid]

Well I'm aware it made it into CentOS but that was kind of short lived and was never updated to newer releases of the codebase (thus making it kind of a init system floating around in limbo with compatibility issues.. many of which I've personally encountered). I understand what you're saying about RedHat vs Canonical, but tooling that is as critical as a containerization or init system don't work very well (IMHO) in a vacuum. You have to have it more widely available, useable and used by the community or not only can you not build community, but you end up creating a fractured ecosystem that makes it hard to tool for among other things.

[wtallis]

Interestingly, upstart's first release was the same month that launchd's license problem went away. If Apple had been a little more responsive to the community, upstart may have never taken off.

[matthewmacleod]

Upstart is a bad example, for reasons that other have pointed out. It appears that LXD is being developed along with LXC, so I can't really see it as a "concurrent solution"; Juju is much the same, in that I don't think it's concurrent with any other project that does that same thing.

So I don't think the comparison with Sony holds any water.

[ambrice]

What about Mir?

[tmerc]

It sounds to me that this is hardware assisted docker containers, which would be a good thing.

Intel processors have VT-x that provides hardware to help speed up virtualization, isolate memory, etc. AMD has something similar. You can break out of a docker container and get to the host OS and other containers. With a hardware assisted hypervisor, it is possible to hide container memory from other containers at a level lower than the "host" os.

If I understand docker and VT-x correctly, hardware assisted virtualization can be used run N instances of a container while only having 1 instance in memory. VT-x can rewrite memory reads/writes transparently and deny writes to certain locations of memory.

Docker containers share the kernel with the host and depend on it for isolation. This would add the hardware assisted isolation of containers without the overhead of another kernel per container, plus the other benefits of docker.

[edwintorok]

So what is the attack surface of LXD? Can the host kernel be compromised from inside an LXD container, i.e. does it provide the same isolation you get when running processes as different users on same kernel, or something more?

For example if there is a local kernel privilege escalation / DoS / etc. bug that can be triggered by a non-privileged user (or a root-inside-container user) will those exploits still run inside LXD?

[tmerc]

A properly written vmm/hypervisor should have no attack surface. This won't be that since there's a rest api on LXD, which is one attack surface.

DoS is still a problem but containers should provide mitigation for that. You can make the vmm prevent DoS, but it's better to keep the vmm small and light.

As for local kernel privilege escalation, yes, it would still run, but it might not matter. In theory, the VMM can isolate all virtual machine resources such that rooting a VM only gives you that VM. I can't figure out how they extend that protection to containers yet since VT-x was made for full virtual machines and containers share a kernel.

[edwintorok]

Well -- 'no attack surface' might be simplifying things a bit too much, as you do need a way to interact with the hypervisor or the privileged host to actually get your data written to disk and your network packets out on the wire. Each such interaction can contain bugs, some of which might be exploitable.

Even with Xen or KVM you do have an attack surface:

* guests can send network packets to the host, which interacts with the networking code on the host. If exploitable you get to execute code/DoS the host. Hopefully not because then so could any other remote machine.

* guests can execute instructions which get emulated / need extra privilege checks done in the hypervisor. See recent vulnerability regarding MSR registers in Xen.

* guests execute hypercalls which obviously interacts with the hypervisor. Bugs here, if exploitable, can be nasty.

* guests need to read/write their data to disk. Are we sure they can't read the data of a (possible already deleted) other VM?

* guests read/write from memory ... was the memory of previously deleted/crashed/migrated guests properly scrubbed? Can any of the hypercalls/etc. be used to read another guest's memory, or access uninitialized memory containing pieces from old guests?

...

Of course the attack surface of a hypervisor is smaller than that of a full kernel (where you also have a lot of syscalls, disk formats, etc.), but that doesn't mean hypervisors are suddenly bulletproof.

I think this quote sums it up quite well: http://marc.info/?l=openbsd-misc&m=119318909016582&w=2. Also have a look at the PDF referenced in that thread.

The question is where does LXD stand from a security pov. between these simplified categories (no order implied):

- running multiple different processes as same user

- running processes in different LXC containers as root-in-container on same host

- running processes in different LXC containers as non-root on same host

- running multiple processes as different users

- running processes in different LXC containers as non-root on same host

- running root processes in different KVM VMs on same host

- running non-root processes in different KVM VMs on same host

- running root processes in different Xen/domU VMs on same host

- running non-root processes in different Xen/domU VMs on same host

- ...

Or in other words if you get an account/container/VM on a shared machine from a hosting provider using technology X, how does that compare to getting an LXD container from a hosting provider? (provided that other unknown users can run LXD containers on the same machine as yours).

[tmerc]

"No attack surface" was definitely simplifying too much, especially for LXD. I think I was trying to say that it is not impossible to mitigate those attacks.

In the pure sense, a hypervisor doesn't need to do anything except create a virtual machine. It doesn't need a way to interact with a user or even the vm once it is created. I have written a bare metal, type 1 hypervisor that did nothing but key log. The guest never made a hypercall and wasn't aware that it was a guest at all. Side note, I'm not an expert. Hypervisor research is just for fun.

We know there is an attack surface on LXD immediately because of the REST API and its interaction with containers. Any resource mediation also exposes an attack surface. Resource mediation is difficult, but not impossible. The attack surface really depends on implementation.

With my limited knowledge of the linux kernel, I can imagine a kernel running in its own vm, a vm for every container, and every container sharing read-only access to the single kernel. Each container could also be isolated via the same memory protection. I don't know enough to say that's possible. I think you're more knowledgeable than I am about lxc and the kernel in general. Any thoughts on this?

[edwintorok]

I'm not worried about memory protection, there is HW support for that and it can be done. I'm slightly more worried about making sure that separate containers can't access each other's disks (via symlinks/hardlinks or overflowing some FS structures).

And I'm worried about the privileged kernel/hypervisor parsing/interpreting data from the unprivileged container. In that sense the situation is not much different from a server: if you can exploit a bug in the server you can run/perform actions with the server's privileges. Same situation with the kernel.

I'd wait until there are some more design/architecture docs about what LXD is exactly to say more though.

[Doji]

Upstart predates systemd by 4 years and had made it into a good number of other distros before systemd took over. Apart from upstart, I see your point.

[cwyers]

There's a bit more to it than that, though. systemd as an init system was originally conceived as a way to improve Upstart; the reason that systemd decided to go its own way rather than contributing to Upstart is because of Canonical's CLA.

That I think is a key point to this discussion -- in order to contribute to an open-source project run by Canonical, they insist upon you giving them more rights to the code you give them than they're willing to give you. Many people are understandably put off by that.

The fact that systemd comes after Upstart is I think less germane to the point that parent was trying to make about Sony than the fact that Canonical insists on being in control of their projects, and puts up rather high barriers to anyone who wants to contribute patches. I am sure that I will hear responses about various patches systemd maintainers have refused or said they wouldn't be receptive to, but that's a difference of kind (not just degree) from insisting upon assigning Canonical the ability to re-license the code outside of the GPL.

[reidrac]

"assigning Canonical the ability to re-license the code outside of the GPL"

I think is more about ownership and IP than re-licensing. Copyright assignment adds value to Canonical as company, eg in case it is acquired.

[cwyers]

These aren't mutually exclusive thoughts; for GPL-licensed content, the ability to relicense the code is pretty much the only benefit that ownership confers over the GPL. It seems strange to assert that having this ability adds value for Canonical in an acquisition, if the acquirer isn't interested in using it.

Now, I have no problem with anyone who wants to sign the CLA and believes that Canonical is acting in good faith. But Canonical is asking for additional value from contributed code than what the GPL provides, and isn't compensating people for this value. Some people have a problem with that, and it makes it harder for Canonical-hosted projects to get community involvement or to be adopted by other distros, where maintainers have to choose between signing a CLA so patches get accepted upstream or continuing to maintain their patches themselves.

[reidrac]

First of all, thanks for the comment. Down-votes don't add too much to the conversation.

I agree with you, but I remember reading that argument backing the CLA (defense in court, to increase the value of the company in case they want to sell it and to be able to close the code); I can't find a link though.

[chimeracoder]

> and had made it into a good number of other distros before systemd took over.

Including Red Hat, which is where systemd was developed!

[diltonm]

I get your points and see it like this, Red Hat is like Ford. Solid, reliable, sometimes quite innovative. Yet the desktop Linux market slipped right by them for like ... years. One has to wonder what could cause such slippage. Ubuntu is more like Samsung to me than Sony. The innovation is off the chart, not everything hits, a lot doesn't but what it does do is cause others to strive and that is leadership.

[olau]

I don't think that's really a fair comment. Red Hat has always focused on the enterprise - home users didn't slip past them, they were never the focus.

The original success of Ubuntu was really incremental improvements - they took Debian as a stable base and improved on the installer and on the defaults, fixing a big bunch of the small obstacles that prevented non-technical people from using the system.

[te_chris]

Say what you will about Sony these days - and there's much to be said - but there is no way Samsung could create something as brilliant as the PS4. Samsung are successful, but I can't think of anything they innovate and lead in, as far as their consumer side is concerned.

[cowabunga]

I think you nailed it. I was trying to articulate this point to a colleague the other day and couldn't do it but that is perfect.

[motoboi]

For me, it seems that Canonical is ever rewriting Ubuntu (the system tools and applications) from scratch.

Last time I saw, they were rewriting it from Python to Vala to fit mobile devices. Before that, they rewrote it from Perl to Python to fit modern development ecosystem.

IMHO, Ubuntu is going round in circles.

They are not pushing their enterprise solutions on Latin America anymore. Don't know what they are doing.

[digi_owl]

At this point systemd has grown way beyond the mission parameters of upstart or similar.