[matthew-wegner]

I'm in game development, and I run ESXi for two reasons:

* Unmodified macOS guest VMs can run under ESXi (if you're reading this on macOS, you have an Apple-made VMXNet3 network adapter driver on your system--see /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext )

* Accelerated 3D has reasonable guest support, even as pure software. You wouldn't want to work in one of those guest VMs, but for any sort of build agent it should be fine, including opening i.e. Unity editor itself in-VM to diagnose something

Does anyone know where either of these things stand with Proxmox today?

I imagine macOS VM under Proxmox is basically a hackintosh with i.e. OpenCore as bootloader?

[wutwutwat]

I run macOS/OSX Sonoma in Proxmox. It does pcie/gpu passthrough (AMD rx580 8gb). The proxmox host is running a AMD Ryzen 7 5700G cpu and has 64gb memory. The mac vm disks are on a zfs fs on a wd black nvme ssd.

It's the fastest mac I've ever owned and it's virtual, executed on a machine running a chip that apple never supported, and you'd never be able to tell it was a vm unless you were told so. kvm and vfio are amazing pieces of software.

A good place to start: https://github.com/luchina-gabriel/OSX-PROXMOX

[rfoo]

> zfs fs on a wd black nvme ssd

Why? IIRC running ZFS on NVMe SSDs limit their performance seriously. With sufficient queue depth modern SSDs can easily get up to 1mln+ IOPS and on ZFS I can barely get 100k :(

[tpetry]

Most probably because ZFS still has the most extensive data correctness guarantess of all filesystems on Linux. Yes, bcachefs will have checksums too but it it is still in beta.

[wutwutwat]

copy on write, instant snapshots, data checksumming, easy rollbacks, auto trim, compression

proxmox takes advantage of zfs' features if available, like when you select block storage on zfs it makes each vm disk its own dataset which you can tune as you see fit for each one. cloning a vm is instant if you make a cow clone. suspending the vm to do snapshot based backups is also nearly instant, etc

Disable atimes and turn off the arc cache and metadata cache (don't really need either with fast storage compared to spinning rust) and use lz4 compression and you minimize write amplification or needless wear on the drive. zfs is perfectly fine on ssd for my use case

[bonton89]

btrfs has checksums as well.

[bonton89]

The lack of 3D paravirtual devices is a real sore spot in kvm. To my knowledge, virgl still isn't quite there but is all there is so far in this space. VMware has the best implementation IMO and everything else is a step down.

[mrpippy]

Note that macOS guest support ended with ESXi 7.0: https://kb.vmware.com/s/article/88698

Running macOS is only supported/licensing-compliant on Apple-branded hardware anyway, and with the supported Intel Macs getting pretty old this was inevitable anyway.

[moondev]

Mac mini 2018 is still the best Mac for vms

   6 cores 12 threads
   64GB DDR
   nvme
   4 thunderbolt3 ports for pci expansion
   10GbE onboard nic
   boots ESXi
   boots Proxmox
   boots or virtualizes windows
   boots or virtualizes linux
   boots or virtualizes macos
   iGPU passthrough
   Supports nested virt

[greggsy]

If you’re just running a bare metal hypervisor then you may as well just go for a second hand tiny form factor platform. No point paying 15-20% extra for the same specs in a svelte case if you’re not running native macOS.

[moondev]

> No point paying 15-20% extra for the same specs in a svelte case if you’re not running native macOS.

The main point is proper non-hackintosh support for running (unlimited amount) of macOS vms on esxi - which requires Apple hardware. Running macOS, Windows and Linux on the metal is another benefit. IMO it's almost as versatile as a framework chromebook (chromeos, crostini, kvm, android with play store)

[mysteria]

The SPICE backend has decent OpenGL 3D support with software rendering, it's slow but it works for simple graphics. It's intended for 2D so the desktop's pretty fast IMO. That only works for Linux and Windows guests though, not Apple ones.

MacOS VMs do work in Proxmox with a Hackintosh setup but you pretty much have to passthrough a GPU to the VM if you're using the GUI. Otherwise you're stuck with the Apple VNC remote desktop and that's unbearably slow.

[zozbot234]

For paravirtualized hardware rendering you can use virtio-gpu. In addition to Linux, a Windows guest driver is available but it's highly experimental still and not very easy to get working.

[zbrozek]

What's the best solution for remote viewing with virtio-gpu? I vaguely recall that it had some incompatibility with spice.

I have a bunch of qemu/kvm virtual machines for various tasks. For two of them I decided that graphics performance needed to be at least OK and ended up buying and passing through old Quadro cards. It'd be lovely to not have to do that.

[mysteria]

I don't think virtio-gpu works on SPICE. I only use it for VMs on my desktop as I can display the output on the local machine using virt-viewer.

[zbrozek]

Still the case, huh. That's unfortunate. All of my VMs run on a headless server, so my only means of use is via some form of remote desktop.

[m463]

You can run macos vms under proxmox.

You can also accelerate video (though it's uncertain what "pure software" means?)

this guy: https://www.nicksherlock.com

has been writing up proxox vms running macos for a while, the latest being:

https://www.nicksherlock.com/2022/10/installing-macos-13-ven...

yes, it uses opencore.

I haven't really bothered with macos GPU acceleration, It is possible but a little fiddly with an nvidia card. I mostly rely on screen sharing to display the remote vm and that's really good (rich text copy/paste, drag and drop files, etc)

As to general guest 3d, you can do this with GPU passthrough, and although it's technical the first time, each vm is then easy.

basically I add this line to each VM for my graphics card:

  hostpci0: 01:00,pcie=1,x-vga=1

and this passthroughs a USB controller for my USB dac:

  hostpci1: 03:00,pcie=1

(this is specific to my system device tree, it will usually be different for each specific machine)

The passthough has worked wonderfully for Windows, ubuntu and arch linux guests, giving a full accelerated desktop, and smooth gaming with non-choppy sound.

two things I wish proxmox did better:

- passthrough USB into containers

usb is pretty fiddly to propagate into a container. It is marginally better with VMs but sometimes devices still don't show up.

- docker/podmain containers

proxmox has lxc containers, but using them is a little like being a sysadmin. Docker/podmain containers that you can build from a dockerfile would be a much better experience.

[moondev]

Nested virtualization also works great on ESXi for macOS guests ( so you can run docker desktop if so inclined ). I believe this is possible with proxmox as well with CPU=host but have not tried it.

For graphics, another cool thing is intel iGPU pci passthrough - I have had success with this when running esxi on my 2018 mac mini https://williamlam.com/2020/06/passthrough-of-integrated-gpu...

[unquietwiki]

* Proxmox can emulate VMWare adapters. https://pve.proxmox.com/wiki/QEMU/KVM_Virtual_Machines

* If you don't mind dedicating the video card to a VM, you can do PCI-passthrough. https://pve.proxmox.com/wiki/PCI_Passthrough

[oneplane]

Apple has been adding a lot of virt and SPICE things IIRC. Some of it isn't supported in VMware (it lacks a bunch of standard virt support), but the facilities are growing instead of shrinking which is a good sign.

On Proxmox you can do the same. You're going to need OpenCore if you're not on a Mac indeed. But if you're not on a Mac you're breaking the EULA anyway.

[rufugee]

Does this work with VMWare Workstation as well? I'd love to run macOS in a VM on my Linux desktop for the few apps I have to use on macOS...

[justinclift]

This works pretty well for running macOS on Linux:

https://github.com/notAperson535/OneClick-macOS-Simple-KVM/

Mostly used it when trying to track down reported macOS bugs for an OSS project, so maybe once every few months. But it's worked quite well at those times. :)

[lathiat]

quickemu also does the job, not just for macOS but many operating systems

"Quickly create and run highly optimised desktop virtual machines for Linux, macOS and Windows; with just two commands. You decide what operating system you want to run and Quickemu will figure out the best way to do it for you." https://github.com/quickemu-project/quickemu

[justinclift]

Thanks, hadn't come across that before. :)

[zerkten]

It has in the past for me, but I haven't run it since 2021.