in your case the target is stationary and the effector is moving so you want to track the arm with respect to the doorknob, the flipside or inverse of tracking a moving object.
https://hackaday.com/2018/07/31/object-detection-with-tensor...
https://github.com/EdjeElectronics/TensorFlow-Object-Detecti...
https://hackaday.com/2020/06/10/autonomous-sentry-gun-packs-...
https://hackaday.com/2018/04/30/training-the-squirrel-termin...