I did that too, and used to worry about it. Then I saw a video of a guy messing around with a giant tub of liquid mercury, talking about how it's not all that dangerous in that form.
As far as I remember, mercury isn't the best thing for you, but it's not as bad as people make out. It takes a fair amount of exposure to do anything too terrible. It's the compounds that become very nasty (eg, dimethylmercury).
Indeed. The equilibrium vapor concentration of mercury is, IIRC, high enough to be hazardous, and you only need a tiny drop to saturate a room.
This was all prominently on my mind as I broke a mercury manometer during an unofficial experiment in school and was trying to find all the little drops that exploded across the floor and hid themselves in the corners. However, I can say that it's a very efficient way to collect dust bunnies... ;-)