You can click `undown` if the downvote is accidental, it remains available for an hour or so after the button is clicked. (I can't remember the exact time, but it does expire at some point.)
Yea, I do that a lot. But I also suspect that sometimes I though I hit the upvote button, but HN registered it as a downvote anyways. Both of the buttons combined are smaller than the tip of my finger, and they both disappear as soon as one is tapped :(