I'm not a hardware guy, but it's been my understanding that this is why drivers exist. Drivers are a nightmare, though. I don't miss dealing with them.
Drivers may provide you semantic uniformity over a diverse set of hardware, but I think Sweeny sounds more interested in performance guarantees, which they may not be able to provide, depending both on the hardware and how well the driver has been optimized.