hm, I'm using X11 forwarding when I'm on my mac box to control my music player and there is pretty much no latency (and text isn't a blurry mess like rdp / vnc but crisp and sharp). I'm on a gigabit network though.
X11 was designed at a time when local networks carried only a few megabits per second. But apps back then didn't depend on sending lots of bitmaps over the wire, just mostly drawing primitives.
Depends on the program! - I've never minded using Emacs that way, even just over wifi, but a lot of more modern stuff is basically unusable. (I assume these programs draw everything to a local buffer, and then copy that to the screen. Probably fine locally, and I'm sure you do get more control, but it's a lot of data to go over a network!)
Local network of VPN? My first attempt at running IntelliJ across VPN last year was using XQuartz and it worked, but latency was terrible and when changing connectivity it would die. Using VNC allowed the process to keep running in the server session over periods of weeks.