Also, X networking was mostly used within local university and company networks, and not over the internet backbone. Well-designed local networks can be very low latency, often noticeably lower latency than disk I/O.
Well, I don't think that network at the time was faster than SCSI, but that's a fair point that X11 isn't quite "internet ready" (though these days with a dedicated VPN you'd probably get very good performance on the core protocol).
To be fair, X doesn't work that well over even local networks with modern apps.
Serving X applications (eg: Chromium) from the Raspberry Pi I'm targeting to the Mac I'm developing on is an exercise in futility, even if it does make 'download this to the target' easier. It's a lot faster to run Safari on the Mac and scp the download to the Pi, just because of interface latency issues.