Then there are a few other physical servers with load balanced redundant VMs, and it fails over seamlessly. HAProxy makes this pretty easy to handle. But we almost never had any hardware issues, servers are pretty reliable.
Actually I think that private cloud-like architecture (a bunch of physical servers with e.g. Dokku running on them) can be a good solution in some circumstances.
Yes, this isn't as much of a gotcha as you make it seem. On prem is literally running your own cloud. The architecture that all the big cloud providers use of racks and racks of servers running hypervisors deploying VMs using a shared storage tier and SDN is the same thing you build when you're on-prem. You are now just the implementor.
Actually I think that private cloud-like architecture (a bunch of physical servers with e.g. Dokku running on them) can be a good solution in some circumstances.