[nonane]

I work on Jump Desktop (https://jumpdesktop.com) that's being used for remote video editing @ 60fps on up to 3 monitors. It supports very low latency remote desktop and even works in environments that have high latency. For example here's Jump connected from the east coast to a physical server in Paris with 150ms ping times between the two: https://twitter.com/jumpdesktop/status/1356679263806763013?s...

It works on Macs and Windows. It works really well with text as well as full motion video.

[astrea]

In your excitement, you also forgot to explain how this is achieved.

[nonane]

Currently we use a modified WebRTC stack with VP8/VP9 to encode the video stream. This is similar to what quite a few game streaming services are doing like Stadia and even Amazon's new Luna(?) service. There are lots of platform specific optimizations up and down the webrtc stack to make sure latency is kept to a minimum while preserving quality. Things like only encoding pixels that change, progressively encoding over multiple frames to sharpen quality, keeping jitter buffers to a minimum.

[fermentation]

Recent advancements in hardware video encoding have made things like this significantly easier to achieve. If you're jumping into a device that has the power for video editing tasks, then it most certainly also has great video encoding capabilities.

[texasbigdata]

Buy it? :)

[dan-robertson]

What’s the actual latency you get between pressing a key and seeing the result on the screen? And how is it if the connection is good and the server is close?

[nonane]

This is a good question. I guess what you're actually interested in is the latency between the physical display updating vs the local display updating, right? I just did a quick test under good circumstances using this video (machines on the same lan): https://www.youtube.com/watch?v=OLxY0HDakRk with the physical and remote display side by side. I noticed an approximately 3 frame lag between the physical display and the local / client display update.

[dan-robertson]

Well what I really meant was the time between pressing a key and seeing a character on the screen. It’s easy to measure (except for deciding when the key is pressed. I go for hitting the key fast and timing from when it stops going down) by using some high speed camera (or medium speed, see https://apps.apple.com/gb/app/is-it-snappy/id1219667593 ) and it’s the latency that actually affects how usable something is for typing. The problem is that it’s only really good for comparing things as lots of devices in the middle add latency, eg keyboards can have 15-50ms of latency[1], typical deferred rendering can give you something like a frame of latency, monitors may add latency or just have slower switching times (it takes longer to switch a pixel to an accurate colour), and some pixels at the bottom of the screen will make it out of the computer 16ms (or whatever a frame is) later than those at the top.

For comparison with my (not particularly optimized) set up going over a fast internet connection, I get something like 150ms between pressing a key and the character starting to show up in emacs (it takes a few ms for the pixels to finish switching). 10 frames feels like a lot to me. My best guess is that without changing anything drastic or eg reducing resolution, I might be able to get that down to 100ms which is still looks like a pretty big number (eg a round trip between London and New York is something like 70ms).

Anyway, thank you for investigating this. A few frames from the network seems pretty good, especially with a reasonably large delta between frames in that video)

[1] https://danluu.com/keyboard-latency/