Or the window manager should zero things out? It seems more consistent for the OS to be responsible for this then say the driver which also does things like supporting OpenGL and GPGPU usecases.
You can generally allocated GPU buffers without the window manager. So to be effective this would have to be in the kernel driver. Opengl/GPGPU is generally done in the user space driver so it'd still be separate.
When graphics card vendors want to protect their competitive edge and not do OS integration beyond dropping horrible and opaque binary blobs every now and then, they'll also have to mop up the security issues such as this on their own.
Technically this should be a OS responsibility, but practically the vendors have made that all but impossible.