[etaioinshrdlu]

So this is pretty misleading. It's really a full system emulator (qemu) running inside Docker, using root privileges on the container that make the isolation very weak (--privileged).

It also uses hardware assisted virtualization (KVM) which is not going to be available most of the time Docker is.

You can think of the Docker platform itself as subset of the Linux platform. With many common features removed by default... SYS_PTRACE, cgroups come to mind as not allowed within the container. (This "Docker as a subset of Linux" is also what you end up getting from most "Docker as a service" platforms offered by clouds, including kubernetes. I'm referring to AWS Fargate, Google Cloud Run, GKE, AKS, here.)

So don't think of this as macOS in docker wherever docker runs.

What would be a lot more analogous to macOS in docker would be running Darling in docker: https://www.darlinghq.org/ ... if that could be made to work for the entire system (highly unlikely)

Darling is more like Wine in that it runs native executables for one platform as native processes on another platform using a compatibility layer. Wine, by the way, definitely works quite well inside Docker.

Also, one final thought. I wonder if you could get macOS to boot in QEMU without hardware assisted virtualization. Then you could probably run this in a fully isolated container again. The performance would likely be abysmal though!

[hinkley]

I don't care though. What I care about is that it's a pain in the butt to do CI/CD pipelines for an application with iOS/OSX support. So if someone has a headless OS X contraption on offer, I want to hear some more about it.

The last time I set this up, a manager decided he wanted a laptop like the rest of us instead of the iMac he got. He asked semi-jokingly if someone wanted the machine for anything and I said "Yes, I do" before he even got the sentence out.

There was just enough memory on the machine for me to set up a few Jenkins agents on it, one for Safari, the rest using the Selenium-maintained docker images.

[dahfizz]

> So if someone has a headless OS X contraption on offer, I want to hear some more about it.

This project relies on OSX-KVM, which is a thing that already exists. The dockerization of that project (what this post is about) is a gimmick as described by GP.

[kogepathic]

I was so excited when I read:

> This Dockerfile automates the installation of OSX-KVM inside a docker container.

Except it automates the fetching of the macOS installation media and launching qemu, which is exactly what OSX-KVM already does. [1] This project does nothing additional to automate the actual installation of macOS inside the VM.

I wish Apple supported installation automation like Microsoft does with sysprep (or Linux with kickstart/preseed). The best I've found is Arduino USB devices that pretend to be a keyboard and mouse to manually advance the installer, which is super lame "automation"

[1] https://github.com/kholia/OSX-KVM/blob/master/OpenCore-Boot....

[xaduha]

> This project does nothing additional to automate the actual installation of macOS inside the VM.

This one does https://github.com/myspaghetti/macos-virtualbox

Not fully, not perfectly, but does. It periodically asks you to press enter and wait.

[dahfizz]

> The best I've found is Arduino USB devices that pretend to be a keyboard and mouse to manually advance the installer, which is super lame "automation".

Definitely not great as an actual process, but this sounds like a super cool project!

[wadkar]

I don’t see any requirement for Apple to provide such functionality. This is simply because all Apple devices come with an OS preinstalled.

And you have tools like Jamf for any enterprise fleet management needs

[jcelerier]

> So if someone has a headless OS X contraption on offer, I want to hear some more about it.

you still legally need to run it on an actual Mac due to Apple's terms of service though... so is there really a lot gained ?

[WatchDog]

If a tree falls in a forest and no one is around to hear it, does it make a sound?

[hosh]

If we are going to divert into Zen koans and non-dual philosophies, then it is definitively "yes it did". The falling away of the perception of an intrinsicly-existing self doesn't change that there's still perception. It would make a sound the same way there is a sound made when one hand claps.

Back on topic though: it's too bad Apple doesn't allow licenses for running things headlessly like this.

[serf]

>Back on topic though

You over-philosophized to the point of bringing their kitschy koan off-topic.

How I interpreted the use of the koan : Apple has no history of legally chasing those who virtualize their operating system; as this is a non-topic thusfar -- who cares?

The hammer may fall one day, but so far 'why are we worried about a legal response that doesn't seem to exist?'.

The answer, of course, is that anyone who builds product based on a legally grey area is at risk when that area begins to crumble.

>it's too bad Apple doesn't allow licenses for running things headlessly like this.

agreed, but I think Apple wants to drive everyone to a hardware solution.

At one point 'enterprise-ish' hardware was offered, but now it seems that it'd be in their interest to offer virtualization licenses while trying to smooth out whatever troubles exist between their software and the major VM hyper-visor offers out there -- mostly since there are huges holes in their hardware offerings for those seeking to do 'enterprise-ish' things en masse.

[microtonal]

Apple has no history of legally chasing those who virtualize their operating system

Maybe not macOS yet. But the definitely chased people who virtualized iOS:

https://www.macrumors.com/2019/08/15/apple-corellium-copyrig...

[klyrs]

I'd interpret that koan to mean that those in isolated woods can make noise without being heard.

Say I have a personal project with a few dozen users. Somebody reports a bug on osx -- I don't do windows, I don't do macs, so I'd need to rely on my small community to fix it. With a pirate copy, I'd be able to do the fix -- and none would be the wiser.

Scale that up to a company, put it up on a public repo's CI, and that's when people might hear the tree fall.

[heavyset_go]

There is if you're running Linux, like you might be if macOS dropped support for your hardware.

[vaxman]

With 40 years of Apple development under my belt, I can safely say that Apple used to be great about supporting older hardware.

When they were on rough times, everyone understood about needing to break compatibility with old hardware (that nobody really cared about anyway because, despite it's horrendous price, was super obsolete due to the rate of Moore's law back then). But nowadays, Apple is invalidating old hardware platforms for superfluous reasons, like abandoning 32-bit apps, enforcing their OEM cryptographic authority (with T2 chip) and getting into it with nVidia (granted nVidia screwed up big time with those GPU chips that blow) or more recently, getting into it with Intel (which has caused agonizing supply-chain issues for Apple). They are no longer that good about helping people support Linux either --you would think that when they deprecate a machine, they would at least have the decency to open source all of its drivers...(and have pre-negotiated the legal rights to do so).

[hosh]

That is exactly what I was thinking of. I don't want to buy dedicated hosts from MacStadium. I got native iOS builds and Unity buidls that would work well with this. Isolation is not as big of a concern.

I too would like to hear if anyone has been able to pull this off.

EDIT: And what about licensing?

[jeroenhd]

Licensing is impossible if you don't run this on Apple hardware. The macOS license only allows you to install and run macOS on devices that came with it and this includes virtualization.

However, if you buy a Mac, install Linux and use it to run the pipelines, that'd be strictly legal I believe.

[heavyset_go]

> However, if you buy a Mac, install Linux and use it to run the pipelines, that'd be strictly legal I believe.

I'm in this boat, so there's a market for it.

[GordonS]

If we put aside the licensing issues, isn't QEMU notoriously slow? Building iOS apps seems pretty slow even on a high-end MBP, so I'd be curious how long it took under QEMU.

Still, very interesting idea!

[znpy]

Qemu Is a bit slow when doing full system virtualization.

Qemu-kvm however is insanely fast as it offloads most of the actual virtualization to KVM and to the native virtualization instruction set .

[q3k]

> isn't QEMU notoriously slow

No! Major cloud providers use it to provide VMs with basically zero overhead.

[tutfbhuf]

Well the title: "A full system emulator (qemu) running inside Docker, using root privileges on the container that make the isolation very weak" would be a bit to long for HN, what do you think?

[etaioinshrdlu]

We are a bunch of pedantic nerds here though. How about "macOS in QEMU in Docker"?

[dang]

Done! Changed from "macOS in a Docker Container".

[AshamedCaptain]

That would not be as clickbaity! Why would you do that?

[tomrod]

Honesty > Clickbait <3

[compsciphd]

its using docker as an artifact delivery mechanism, not as an isolationism mechanism.

[ronsor]

Honestly that's kind of yikes. That probably just adds extra complexity, and there are better ways to deliver VMs.

[paulfurtado]

If you have container orchestration in place, being able to use it to run VMs via qemu is actually incredibly useful and isn't really much of a yikes. Sure, you're losing the container's isolation features, but you have a VM there, which is even stronger isolation.

We do CI for our VM images in our kubernetes clusters. The build system already was in kubernetes, so putting the OS image testing in there was a big win.

The benefit of doing this is also that on a personal machine you can start playing with an OSX vm with a single docker run command with no other dependencies and many people already have docker setup, whereas standardized qemu/virtualization tooling is now much less common on developer machines

[dahfizz]

I don't think you understand what this project is.

You have to fiddle with BIOS and kernel module parameters, install packages, configure KVM, etc on your docker host for this to work. It's not something that you can just throw into kubernetes, especially if you don't manage the kubernetes deployment yourself.

> The benefit of doing this is also that on a personal machine you can start playing with an OSX vm with a single docker run command with no other dependencies

There _are_ external dependencies that you have to set up manually. It's the same amount of work to set up on docker or to use a real VM, so I can't imagine why you would prefer this method.

[paulfurtado]

To run 64 bit VMs, you always needed to turn on hardware virtualization in the bios. To configure kvm, all you need to do is "modprobe kvm" if it isn't already loaded. At that point, everything else is user-space and 100% of the user-space dependencies are installed in the image. All the docs about libvirt on the github page are unnecessary. So full steps are really:

1. Enable hardware virtualization 2. modprobe kvm 3. docker build 4. docker run

and you have an OSX VM.

> It's not something that you can just throw into kubernetes, especially if you don't manage the kubernetes deployment yourself.

GCP and Azure support nested virtualization, so you actually could do this in a managed kubernetes cluster. It's plenty common to use privileged DaemonSets in kubernetes to load kernel modules for filesystems, storage, or iptables rules. If you're allowed to run privileged containers, it's trivial to run VMs like this in kubernetes.

[slimsag]

I work at Sourcegraph and have been considering something like this for a while for running long-running jobs, things like CI pipelines and GitHub actions for example.

How would you feel about an app like GitLab, for example, shipping a docker container that required privileges for this, I wonder?

[paulfurtado]

It'd definitely be a harder sell for third party software to require a privileged container and /dev/kvm mounted when you run it in your environment, especially since nested virtualization is largely unavailable in AWS. It also requires that the correct kvm kernel module is loaded, etc.

However, if it was a product that required virtualization and that was recognized as a requirement, then also distributing a docker image that could do it would probably be useful for people in the "and if you don't have virtualization infrastructure, but have container orchestration and nodes that support virtualization, our service will also work in a privileged container" camp

[slimsag]

Good points, I appreciate the response!

[MiroF]

> there are better ways to deliver VMs

For the curious - what are they?

[flatiron]

Not op but they probably mean vagrant?

[brutal_boi]

0PeNStaCk

[wadkar]

I would say docker is primarily used as an artifact delivery mechanism where artifact presupposes inclusion of required runtime.

I an not saying it’s right or wrong. I think it’s the most practical and thus most pervasive use of docker.

The company and the project provided a great service by creating lot of shipping/container related metaphors though. That’s very much invaluable in my book. Providing thousands of developers means to talk about objects that are static and dynamic from an architectural perspective.

[saurik]

(Darling requires a kernel module, which also isn't a thing you are able to do in the context of Docker as you are often just working with the host kernel.)

[etaioinshrdlu]

Wow! This is highly surprising and is totally unusual! They're embedding parts of the XNU kernel in Linux. I can imagine this was done as a last resort as there is pretty much no chance this will ever get upstreamed, not just because of licensing, but also simply just strangeness.

[userbinator]

I have run MacOS in QEMU in emulation mode, that was around the time of the first Hackintoshes. You're right, it was very slow (on the hardware of the time.)

[nunez]

It's also been done before: https://github.com/kholia/OSX-KVM

[m463]

(that's referenced right there in the readme)

[m463]

I think "docker" is an ambiguous term nowadays.

Depending on your point of view, it could denote a container, or source control, or a sort of makefile.

To me it is a reproducible recipe in 43 lines of Dockerfile text.

[AshamedCaptain]

This thing is basically a bash script that runs the package manager of a distro pointing to the latest package feed; not even a specific stable version or anything like that.

It also depends on host kernel features such as KVM settings and hardware such as CPU (good luck on AMD) much more than whatever is packaged on the container.

This is about as "reproducible" as replaying my .bash_history on a different machine. I would bet that before the end of this year this script no longer works.

[chooseaname]

> I think "docker" is an ambiguous term nowadays.

Why do we do this to ourselves? Words have meaning. This should go doubly so for engineers. Our language should be as precise as possible when describing systems.

[akimball]

English vocabulary rules - vocabulary is built of axiom schemata - are insanely complex, and still we have to rely heavily on disambiguation by context. It is the nature of NL semantics. I'm not arguing against more precise terms of art, just saying that expanding an insanely huge and complex vocabulary isn't a great way to accomplish that. Establish your context, then use your symbols in that context; thus are all utterances disambiguated.

[sigjuice]

It is definitely a recipe, but the "reproducible" claim is weak. These 43 lines of Dockerfile text are unlikely to work in a few months (or even weeks).

[yjftsjthsd-h]

Okay, I know how docker can mean containers, and I agree that Dockerfiles are quite similar to makefiles, but how does docker relate to source control?

[m463]

Docker keeps a history of changes to the container with layers, and you might arguably treat it like version control.

maybe "docker history" ~ "git log"?? tags=branches?

Docker isn't one thing, so it's ALL grey. It's not really make. It's not really a vm. It's not really even reproducible.

One thing is for certain - it's hard to describe docker to new folks so they get it. "it's sort of like netflix, for cows, but in space..."

[Denvercoder9]

That's the most insane use of Docker I've ever heard of.

[m463]

I'm not saying people use it for vc, it's how docker works. It optimizes your containers using layers and if you do "FROM ubuntu:18.04" in 3 dockerfiles, you only have one copy of the ubuntu stuff.

[fulafel]

KVM will be available if you run Docker natively on your laptop, or if your docker-on-VM setup supports nested virtualization - I think these are pretty common setups?

[mister_hn]

yes, you can do nested KVM virtualizations, you need to export specific CPU flags

[fulafel]

You can run inside any hypervisor that supports nested virtualization (eg hyper-v, vmware esxi, etc in addition to kvm).