I disagree. Just because I downvote a post doesn't mean I want it to disappear. I may want to revisit the discussion at a later time, in which case I may want to review everything that's been said.
Read my other response, my intention was about whoever downvote a comment not a submission. So you would be able to see the rest of the discussion about any submission, I made a mistake omitting the comment restriction.