[R0b0t1]

Hm, I've actually thought about that a lot. I'd probably keep the bytestream format. It is insanely useful. However I might open up multiple streams. Stream one is basic IO. Stream two could be formatting info. Streams above that implementation dependent, but you could probably start defining them for keyboard/mouse/HID IO or audio and video. Then to make it work over the internet just shove it over SSH or TLS.

The one downside that has is if you separate the formatting from the data you can not recreate the session without keeping time info somehow. So, might need a fix for that.

The only other real alternative I'm aware of is moving the formatting into the API "stream" that communicates using the machine ABI. This is a terrible solution. It's what Windows uses.