I've long held that you should have to reply to the comment when downvoting. Even if it's just a "no", that's better than the ability to hide a comment you don't agree with
The problem is that extremely low-quality comments (e.g. anything from "agree" and off-topic memes to irrelevant political sledging to the kind of thing that will get the user banned when a moderator sees it) are the majority of downvoted comments. Requiring a reply would give them long chains of unnecessary comments until they hit the threshold to be hidden.