In a desktop, couldn't you assign a GPU to one video machine and in that scenario would there still be a security problem when there is only one VM using it?
It’s not about virtual machines. GPUs typically have direct memory access to pretty much all system RAM. There exist PCIe mitigations, but the review does not meet up to Qubes security standards.