[burke]

As someone working in developer tools for a company with thousands of people developing software on MacBooks, MAN do I resent SIP. I've recently started calling it "Systems Implementation Prevention".

It's incredible that it's 2024 and I can't cobble together anything vaguely container-like on macOS because:

* bind mounts don't exist (?!)

* clonefile() could maaaybe do the job but doesn't work cross-volume and a lot of the stuff outside of /Users is a different volume

* there's no filesystem namespace.

* chroot doesn't work either because /usr/lib/libsystem.B.dylib is required, but also pretend.

* And it sounds like chroot runs afoul of some SIP rule nowadays even if you can get past the above.

* A lot of this could be worked around with FUSE, but in order to turn that on, we'd have to turn off a lot of SIP.

The closest we can get without virtualization is sandbox-exec, which just allows allowing/denying file reads by path, with no path translation. And also is deprecated.

Nevermind that dtrace exists but you're not allowed to use it either.

Truly, the worst UNIX.

[ChocolateGod]

> * A lot of this could be worked around with FUSE, but in order to turn that on, we'd have to turn off a lot of SIP.

Didn't the latest MacOS update add something similar with userspace filesystems

[burke]

File Provider: https://developer.apple.com/documentation/fileprovider/

Interesting, I hadn't heard of this. First impression skimming the docs is that they've gone to significant trouble to make it not generically useful as a FUSE replacement but I could be misreading.

[mbirth]

Not macOS directly, but there’s fuse-t which works in userspace and just creates an NFS server which it automatically mounts via macOS-own capabilities.

The library is a drop-in replacement for libfuse and works great for me.

https://www.fuse-t.org

[viraptor]

> Nevermind that dtrace exists but you're not allowed to use it either.

You're not losing anything, dtrace even without SIP has been broken and unusable for at least 2 major versions now.

[pxc]

What's broken in dtrace even with SIP disabled?

[jsolson]

Why do you want to avoid virtualization?

[burke]

It's very heavyweight, and there's no good shared filesystem option.

We did use virtualization for a bunch of stuff before the move to Apple Silicon, back when Hypervisor.framework and xhyve actually existed and were plausibly useful.

Those also fell by the wayside in the architecture migration and now virtualization has a massive performance cost.

Apparently the M4 chips are on ARMv9 which is apparently much better at virtualization, but it remains to be seen whether apple provides anything lightweight again.

[plorkyeran]

Filesystem perf is definitely an issue, but cpu wise virtualization perf is basically free on Apple Silicon. I don’t know why you think Hypervisor.framework went away or became useless in the architecture migration. Obviously x86 VMs are slow, but we’ve been using arm64 VMs for years now with great results.

[burke]

Ok, probably I'm over-interpreting some first-year pain with the new chips.

Anyway, even at that, virtualization isn't really my preferred way to solve any of the problems I have.

[darklion]

Uh, Hypervisor still exists, and is still supported: https://developer.apple.com/documentation/hypervisor

[burke]

Yep. However, before the Apple Silicon migration, VT-x gave us extremely low-overhead virtualization. We built a tiny linux kernel that booted in a second or two and were able to run whatever we wanted with minimal perf overhead.

In the Apple Silicon migration, obviously emulating x86_64 got slow, but even when we built ARM64 VMs, performance was still miserable: there was (is?) no way -- at least no way we ever figured out -- to get reasonable perf out of virtualization on a macbook.

It's possible that this changed post-M1 and it sounds likely it's set to change with M4.

EDIT: ok, I'm probably hallucinating more problem than there actually turned out to be based on the pain in the first year of the M1 chips.

[inkyoto]

If you are referring to the nested virtualisation support in ARM v8, it was added in the ARM v8.3-A revision of the architecture, and M1 uses ARM v8.5-A as the baseline.

But yes, virtualisaiton support for ARM (in general) was abysmal and Apple Silicon was the catalyst that pushed people over the edge towards improving it across aarch64 (also in general).

[hi-v-rocknroll]

> It's very heavyweight

I guess your dainty, utopian senses are irrationally offended by something that works. Some types of virtualization offer hard isolation guarantees while cgroups, chroot, jails and the like provide pretend isolation lacking hard guarantees about either security or resource limits. KVM is tiny and so is Virtualization.framework. If you want perfect "containers", you're not going to find them anywhere because they try solve a problem (convenience, speed, and isolation) at the wrong level, in the wrong way. Type 1 Xen and VMware are the gold standards supporting all sorts of deduplication, replication, and migration options that containers can't touch. Type 2 Kata Containers is another option out there with stronger guarantees with the same interface as CRI. If these don't work for you, write a better solution that can divvy disk IOPS and latency, process manipulation, memory shares and bandwidth, network bandwidth and priority, and VFS fairly while sandboxing misbehaving processes from taking down other containers on the same host. I submit that these are essentially impossible goals with the architecture of Linux, which is why variants of virtualization providing paravirtualized guests is generally superior in providing service guarantees because there is out-of-band management exterior to fallible, DoSable containers.

[burke]

> I guess your dainty, utopian senses are irrationally offended by something that works

1) What

[talldayo]

> If these don't work for you, write a better solution

It's MacOS, you fucking can't.

[Vilian]

Bro read one line and went crazy

[m463]

> Truly, the worst UNIX.

You're not the target market.

:(

[Vilian]

It's still the worse unix, if the target market don't care don't change that fact

[talldayo]

If the Xserve's legacy is anything to go by, neither are UNIX customers.