This would make a lot of sense. Flagging clearly says "this comment is not appropriate on this site", while downvoting usually means simply "I don't agree with this comment".
Downvoting also means "this comment should be this high", however the problem is that there's o indication when a comment is at score of 1, so people would inadvertently push it into gray area.