[kopo]

It's not simple to do.

First you need a whole lot of data to train the system which doesn't really exist (unlike the case of speech to text or one language to another(text to text)). You would need to set up a big data collection project.

Secondly there are a whole bunch of different things to track. Movement of multiple fingers, +palm, +entire hand, +facial expressions. Current state of the art would be tracking a single thing like a ball or a player for sports and that still requires a number of cameras and a couple people sitting behind the scenes fixing issues.

I think Microsoft's Kinect came close to achieving something akin to sign language recognition at a very basic level. But from what I remember reading they spent a whole lot of time and resources training their system just to get there.

Maybe 3-4 years away I'd say.