I do this too. Particularly when I'm reading something online or thinking. I was smoothing my sideburns whilst reading this post, looking for "that special hair". I favour my sideburns and beard.
Heh, I was just doing it with my eyebrow. Noticed a harder, thicker hair among the softer hairs. Pulled out five other hairs before getting it. Probably have a hole in my brow now.