[Jugurtha]

Thought experiment: play a video and close your eyes. You will know when the ball's in play by ear.

I think the best bet would be sound event detection. What's associated with players actually playing? Both sounds of ball/racket and ball/table contacts and impacts.

From a whole video, you cut out the segments where there's no impact plus a few seconds (end of play) and minus a few seconds (before serve).

A crude approach/proof of concept would be a simple (sound) peak detection which could go a long way in my opinion.