That wouldn't be too hard if GPU's would have a stable interface. Try programming a GPU in Assembly language and see how that goes. The experience sucks, but that's the level that needs to be targeted in case of an OS.
For example, in the past Amiga computers had a 'GPU' (although much less powerful than todays GPUs) with a stable interface. It was a first class citizen in its OS. It also was incredibly easy to target in Assembly language.
If programming GPU drivers was not something only a handful of employees with NVIDIA or AMD badges could do (because of NDAs, non-public documentation and immense complexity), somebody would have tried.
Writing GPU drivers is not hard per se. It is impossible if you don't work at the vendor and have access to internal documentation.
Knowing how to write a kernel (which incidentally I am doing for the second time) doesn't mean you have years to dedicate to reverse engineer something as complicated as a GPU