Depending on your VM you'll need to enable GPU passthrough (sometimes erroniously labeled "3D acceleration" or something), and then install the corresponding GPU drivers in the guest VM.
Hmm, in theory what if I used a dedicated GPU just for the VM that was disabled on the host. I don't know the space of these exploits well enough to know if there's an obvious attack that still leaves the host open to.