I agree, downvoting is a bad idea across the board for silencing people that are not in the majority group-think. You don't even have to have a reason to down vote someone.
What I find interesting is that on HN, downvoting doesn't seem to have a negative impact, not as much as Reddit anyway. And I wonder if that's because folks who use HN really care about HN - I do - and make a good-faith, concerted effort to contribute in a positive way, with moderators who will enforce rules, but will not remove posts they disagree with (Reddit), or mandate arbitrary and inconsistent standards (Reddit).