Both Windows and OS X provide ways for a sandboxed process to draw directly to a GPU-resident texture owned by another process. Chrome's renderer sandbox is a good example of this technique.
A media player is rarely that simple. Before writing to the GPU you have to access the content (file, network), demux, then decode and finally display.
The most dangerous areas are the demuxers and decoders so they have to be sandboxed.
So yes, you're right, but this doesn't solve the problem of moving the buffers between processes.
You can co-opt Chromium's. I've been doing that with a project I'm working on and it has worked quite well.
I'm also not sure where you'd lose much performance. If you hand the file handles/sockets and backbuffer to the renderer, you only need enough IPC to synchronize the drawing. Sending small messages on the order of 100 times per second between processes is not going to be a bottleneck.
But you need to pass data from the access to the stream_filter, from the stream_filter to the demuxer, from the demuxer to several decoders, from the decoders to potentially a few video-filters and chroma-converters, and then finally to the output. Each of them need different access policies, and several of them require FS access.
It is not easy. If it was, people would have done it already.
You can get most of the security people are asking for without that level of granularity. If the decoder is exploited and is able to attack the demuxer, that's still worlds better than the decoder being able to run with full user privileges.
Do any of those components need unrestricted/unpredictable file access? Because if they don't you can just open the files in the main process that handles the UI and send them to the sandboxed process via IPC. None of Windows/OSX/Linux do permission checks when file handles are read from, they only check when the file is initially opened.