[phoboslab]

This is really cool!

The "video" quality is pretty bad though. From what I can see, the app streams base64 encoded JPEGs via WebSockets and draws them in a Canvas. The overhead of sending complete images costs a lot of bandwidth.

You may want to look into jsmpeg for live streaming: https://github.com/phoboslab/jsmpeg

[TheMakeA]

We've actually experimented with jsmpeg for this type of thing. The problem we ran into was that jsmpeg seemed to require every frame from ffmpeg. If the browser or network slowed, the feed would fall behind the simulation, leading to several seconds of latency when interacting.

We'd love to use it and appreciate your hard work though. Do you have any ideas on how that problem can be solved?

[phoboslab]

There are two sides to this problem. For jsmpeg (or even in the server), it would be possible to simply wait for the next intra frame (a frame that doesn't require previous frames to decode) on certain conditions and throw away all other frames in between. This will give the decoder and network time to catch up. By default ffmpeg emits one intra frame per second, iirc.

The more difficult part is probably detecting this lag. I'm not sure how this could be done. Maybe the node websocket server (ws) can detect congestion somehow.

[TheMakeA]

Thanks. Maybe our ffmpeg settings weren't creating intra-frames somehow (or we happened to drop every single one) because the stream would never recover. I'll probably try it again at some point soon. We really want jsmpeg to work.

[userbinator]

base64 encoded JPEGs

MJPEG is supported in all browsers I know of, and would cut down on bandwidth by 33% over base64.

(If you really want to use WebSockets, a more dense encoding than base64 could also help - look into base85, for example.)

[niutech]

MJPEG is good, but keep in mind that Chrome doesn't support playback of MJPEG streams as a main resource, it has to be in an <img> tag.

[userbinator]

Looks like that functionality used to be there, was broken, fixed, and then broken again sometime later. The latest bug for it appears to be https://code.google.com/p/chromium/issues/detail?id=308999

For this application you'd need it to be in an <img> tag anyway since there's the other stuff on the page too, so that's a moot point.

[niutech]

Not only <img>. The MJPEG stream could theoretically be embedded in an <iframe>, thus being affected by this bug.

[kolencherry]

Why not use WebRTC for streaming the video?

[TD-Linux]

You don't even need to go that complicated. A WebM stream over HTTP is pretty easy and would work well for this.

[cbhl]

It's worth noting that Chrome Remote Desktop uses VP8 (which is the video codec used in WebM):

http://googleblog.blogspot.com/2012/05/next-step-in-chrome-o...

If you're having performance issues with it, you just have the video codec tuned wrong.

[weiyin]

definitely a possibility :)

we'll check that out, thanks for the heads up.

[weiyin]

we did some testing using compressed video, but ended up finding single frames to be more performant. also, the setup can handle much higher framerate and quality but we've kept things fairly lean for the demo :) can easily handle 15fps w/ high quality images. thanks again for the questions and comments.

[weiyin]

Thanks for the suggestion, that looks like a cool library. A future goal is to be able to provide higher resolution images to fully show off the high DPI displays.