That’s true, bhyve doesn’t support nested virtualization right now.
In practice though, most setups don’t actually need it if you’re running workloads directly on the host.
Also, if your goal is testing or simulating clusters, you can already run Sylve inside jails. That gives you multiple isolated “nodes” on a single machine without needing nested virt. We have a guide for it here:
https://sylve.io/guides/advanced-topics/jailing-sylve/
So you can still experiment with things like clustering, networking, failure scenarios, etc., just using jails instead of spinning up hypervisors inside VMs.
Nested virt is still useful for specific cases like testing other hypervisors or running Firecracker inside VMs, but for most Sylve-style setups it hasn’t really been a blocker.
We run Proxmox VMs that are running Hashicorp's Nomad orchestration at $DAYJOB. The Nomad clients are then turning around and running the docker containers (Proxmox -> Nomad VM -> Docker). For us it's easier to manage and segregate duties on the initial metal this way.
Nested virtualization can be very handy in both the lab and in production. In the lab, you can try out a new hosting platform by running one atop the other. IE: Proxmox on VMWare, Hyper-V on KVM. This lets you try things out without needing fresh bare metal hardware.
In prod, let's say you run workloads in Firecracker VMs. You have plenty of headroom on your existing hardware. Nested virtualization would allow you to set up Firecracker hosts on your existing hardware.
Perhaps I'm misunderstanding, but wouldn't that case be covered by simply putting some vms under a vnet and others on another vnet and make them talk to each other? I can't also understand what you mean by "fresh bare metal hardware". In either case you don't need bare metal, being a top level vm or a nested one.
If you're evaluating VM hosts (proxmox, hyper-V, vmware, etc...) You need to have support for nested virtualization all the way down. Otherwise, if you want to evaluate a VM infrastructure, you need to start with bare-metal. Really, you just need to make sure that your top level support nested virtualization, but I understand their point.
However, the point about firecracker VMs in place of containers I think is really a good use-case. Firecracker can provide a better isolation environment, so it would be great to be able to run Firecracker VMs for workloads, which would require that the host (and the VM host above) support nested virtualization.
One example: when learning Proxmox itself. I was able to set up a multi-node cluster with more complicated networking than I was normally comfortable with and experiment with failures of all sorts (killing a node, disabling NICs, etc.) without needing more hardware or affecting my existing things.
Outside of learning and testing I am not sure of what uses there might be but I'm curious to know if there are.
In practice though, most setups don’t actually need it if you’re running workloads directly on the host.
Also, if your goal is testing or simulating clusters, you can already run Sylve inside jails. That gives you multiple isolated “nodes” on a single machine without needing nested virt. We have a guide for it here: https://sylve.io/guides/advanced-topics/jailing-sylve/
So you can still experiment with things like clustering, networking, failure scenarios, etc., just using jails instead of spinning up hypervisors inside VMs.
Nested virt is still useful for specific cases like testing other hypervisors or running Firecracker inside VMs, but for most Sylve-style setups it hasn’t really been a blocker.