I assume your concern with GPU passthrough is that each VM needs a whole GPU?
You can use GPU-PV to split your GPU between VM instances.
Then the main bottleneck becomes how thin you split out your VRAM.
Wouldn't virtualbox or vmware's paravirtual GPUs be a better fit for this use case? Unfortunately the offerings with qemu/libvirt still lag vmwares by a lot.
I know those offer virtual GPUs, but I am unfamiliar with any paravirtual GPU offerings from VMWare or VirtualBox. The virtual GPUs are much more limited in performance and graphics API support.