There needs to be a separate mechanism for disagreement versus quality (affecting ranking). Both of these should be separate from flagging for moderation.
There is. Replying. If there already is a disagreeing reply, you can upvote that. Downvoting a constructive and well-written post just because you disagree is not justified.