Luckily the video processing libraries provided by Intel/NVIDIA/AMD are cross platform (Win32/Linux). So the arm specific video processing code for the RPI is a different can of worms, the rest of the client (window creation, polling for input, etc) should be able to support Linux and X11 generally.