[sgarland]

Yes. I first tried it with Rook, and that was a disaster, so I shifted to Longhorn. That has had its own share of problems, and is quite slow. Finally, I let Proxmox manage Ceph for me, and it’s been a dream. So far I haven’t migrated my K8s workloads to it, but I’ve used it for RDBMS storage (DBs in VMs), and it works flawlessly.

I don’t have an incredibly great setup, either: 3x Dell R620s (Ivy Bridge-era Xeons), and 1GBe. Proxmox’s corosync has a dedicated switch, but that’s about it. The disks are nice to be fair - Samsung PM863 3.84 TB NVMe. They are absolutely bottlenecked by the LAN at the moment.

I plan on upgrading to 10GBe as soon as I can convince myself to pay for an L3 10G switch.

[sixdonuts]

Just get a 25G switch and MM fiber. 25G switches are cheaper, use less power and can work with 10 and 25G SFPs.

[sgarland]

The main blocker (other than needing to buy new NICs, since everything I have already came with quad 1/1/10/10) is I'm heavily invested into the Ubiquiti ecosystem, and since they killed off the USW-Leaf (and the even more brief UDC-Leaf), they don't have anything that fits the bill.

I'm not entirely opposed to getting a Mikrotik or something and it just being the oddball out, but it's nice to have everything centrally managed.

EDIT: They do have the PRO-Aggregation, but there are only 4x 25G ports. Technically it _would_ meet my needs for Ceph, and Ceph only.