Apple's neural engine shows that you can live in between those two worlds.
As you said, the trouble is the latency, the programming model is still great.