Hacker News new | ask | show | jobs
by nick__m 5 days ago
Before my vacation we (3 colleagues and myself) completedan 8 months long migration (coordination with stakeholders is longer and more complex than migrating a 192TB VM !!!) to 6 proxmox clusters so 20 to 40 clusters for 40k is certainly possible but imo it would be unwieldy.
2 comments
throwaway270925 5 days ago
> 192TB VM

Why? Honest question, what leads to that Kind of size and why cant it use NAS shares or SAN disks for most of that data? Kudos on the migration!

link
nick__m 4 days ago
I wish I knew ... at least it was a span volume so we could use proxmox support for vmdk and achieve about 30 minutes of planned downtime but a week of storage vmotion followed by another week of the proxmox equivalent.

When I get back to work I will finish the samba configuration of the ceph cluster front-ends to replace those elephants.

link
SV_BubbleTime 5 days ago
Would you do it again (ProxMox)?
link
nick__m 4 days ago
The FC multipathing was a learning experience and the manual workload placement requires good metrics on your workloads. The built-in ceph is decently configured and more performant than FC if you have 100GBS mellanox NICs and adequate quantity of ram. Veeam integration is serviceable but it's not as mature and polished as the integration available on vmware.

Having tried azure local before (it seems magical but the more you use it the worst it gets, update failed for no apparent reason on only some supposedlyidentical nodes, the sdn was atrocious to deploy and was manageable from wac only), I would recommend proxmox over it anyday.

If you don't have linux expertise on hand and have traditional FC based storage, I would recommend something else, probably nutanix if your budget is big enough.

link
SV_BubbleTime 4 days ago
And if you had to do it one a normal human setup of three hosts and 15 total VMs, no fiber, no san, no special nics or anything like that, just tied in to an ordinary Veeam with the ssd storage on an immutable Linux machine elsewhere?
link
nick__m 3 days ago
I would choose promox or ovirt (oracle virtualiztion manager is pretty nice, I use it because the licence of the only oracle database of our institution requires it, that and a specialc cores pinning configuration, to avoid licensing every cores in all clusters, https://houseofbrick.com/wp-content/uploads/2015/03/Oracle-P... ) but it's oracle so we went with proxmox.
link