Oh, I missed your comment about being able to tune some wristwatches quartz! I wasn't aware that was a thing.
Still, wouldn't the temperature of a watch while being worn vary as least as much as when sitting in a drawer (unless you live in a region blessed with t-shirt weather year around)?
One of my favorite wristwatches I used to wear as a teenager had a thermometer, but I don't remember how exactly that varied over the year, just that it always showed neither quite my body temperature, nor quite the ambient one :)
The crystals used in watches are usually cut and selected so that a local minimum or local maximum of the tempco is near the temperature of your wrist.
Thus, the tempco is near zero, so human-to-human differences don’t matter much.
One thing to notice is that quartz watches almost always have a metal backplate touching your wrist so that the crystal can have good thermal contact. Presumably, the thermometer in your watch was decoupled from that plate.