Not as secure as VMs but GPU passthrough with Docker/Podman is much easier to set up, and you can even use the GPU on the host machine at the same time.
Nvidia provides a toolkit to do this [1], getting a GPU into a container is as easy as running `podman run --device nvidia.com/gpu=all`. The process is similar for Docker, but rootless Docker requires some extra steps IIRC.
(Would you even need anything fancier? I think /dev/dri is supposed to isolate users.)