I think they count, never really checked specifically though and the karma system isn't exactly spelled out in detail anywhere. I'm pretty sure your downvotes to the parent and children of a comment you make don't count though - or at least I read that a few times while looking into the site initially.
I bumped your comment up, if you want to find out I should be able to un-up it once you check your current number... if I remember to refresh my comment history ;).
It's probably a good thing points aren't the main thing.
HA that's exactly the main reason I keep an eye on it up top to know if I should check my comments to respond to someone. I'm at 1631 right now but also a couple other recent comments in the mix. Though also with that lately I've noticed more random downvotes for what I don't see any reason for.