[alanctgardner2]

I've done basically this (technically RFB is the protocol VNC uses, but whatever). The team I was on makes an auditing tool that records RFB traffic, transcodes it into MPEG video and also does real-time compositing in RFB. Initially it was just static boxes to obscure stuff, but we started doing messaging as well, until eventually we had a library to write arbitrary strings to the screen. I always wanted to extend it to accept user input as well, but that ventured into X territory too much.

It's worth pointing out that adding new data to an existing RFB stream with any kind of speed is stupid hard. The server has the ability to send one of about 11 types of message, from simple - a bitmap or RLE - to stupid - hextile and tight are popular. The only way we found was to parse every message, update a frame buffer, then re-encode the updated framebuffer. Not to mention all of the clients and servers have slightly different implementations, so even though you should be able to implement a subset of the spec, you end up implementing the whole spec, plus kludges for every popular client and server. Particularly egregious is Jolly's Fast VNC, which is probably the best Mac VNC client, but it actively rejects servers which are within spec to accommodate one particular server the dev targeted.

/rant

If you want to see the actual RFB spec: http://www.realvnc.com/docs/rfbproto.pdf