This will happen, of course, but shouldn't last long. Also momentum of the tracked object should help.
Btw, funny that my comment above attracted so much hatred. Didn't want to sound like a know-it-all but yes, at least in the case of the referee I guess that checking for a body would have helped.
For example, in this stylised picture we can easily tell the player's head from the ball:
