But when I go look at my text editor being slow, I can see that the amount of CPU time spent dealing with the kernel is less than a tenth of it. So that's not the reason.
It's a much better estimate than hand waving about memory isolation.
If we want to talk about how things work directly, my program can get things to the GPU in far less than a millisecond. The safety layers are not the problem.
Dan Luu did cool experiments with input lag (https://danluu.com/input-lag/).