I used to really want downvoting. Flagging really works much better. The stories that I would have ended up downvoting either don't belong here and get [dead]-ed, or they spawn a lot of fascinating conversation.
Fortunately, people haven't been doing a lot of gaming of the system yet. The biggest problem with upvoting only is that controversial posts garner the most votes (a thousand people love a subject and ten thousand people HATE it, so it gets a thousand upvotes and a TON of comments, even if most people disagree with the premise).
I agree, downvotes are dangerous. I know Reddit had (has? I don't know I left when I found HN) a problem where all new articles were getting downmodded by people who wanted their own links on the front page.
I repeatedly think that I would like downvotes, but I quite quickly remember that I absolutely hate downvotes. I don't completely agree with being able to downmod comments, however I think the side effects are dampened that you have to have good karma before you can downvote, which prevents trolls from abusing the system.
Fortunately, people haven't been doing a lot of gaming of the system yet. The biggest problem with upvoting only is that controversial posts garner the most votes (a thousand people love a subject and ten thousand people HATE it, so it gets a thousand upvotes and a TON of comments, even if most people disagree with the premise).