I'm thinking of making my next desktop with a support for giving a virtual Windows machine access to a video card -- near native performance, but Linux for the host OS.
Yup, I'd strongly consider something similar. But I'd have to seriously consider the security implications: giving Windows access to the graphics card could end up being more hassle than it's worth.
GPUs' memory management isn't exactly security friendly.
-> https://arxiv.org/pdf/1305.7383.pdf