Author here, this is a known issue and being worked on for handtrack.js 2.0.
The face/head errors is likely an artifact of transfer learning (handtrackjs is finetuned on the an object detection model trained using the coco object detection dataset which contains a person category).
Current approach being explored to expand the training data to include faces/heads such that the model better discriminates this from hands.
There are also size and speed optimizations being explored (smaller models such as efficientdet, quantization).
It also recognizes my ceiling as a hand when there's no hand in front of the camera. Not enough negative training data?
To satisfy the curiosity of VoidWhisperer: the score was 0.868. And it also sometimes loses track of a hand, in my case quite reliably when it's flat and seen from the side.