[douche]

I don't believe that you can downvote a comment that you reply to - so you would have to downvote first, and then reply.

[detaro]

other direction: you can't downvote direct replies to you. You shouldn't see a downvote button on my comment, but on the comment above you.