I prefer the suggestion to show a percentage instead of the raw count of upvotes and / or downvotes. Seems more meaningful and contextual. And less likely to contribute to a mob mentality like behaviour (where people in an online community tend to be swayed by the majority opinion and sometimes don't post opposing views because they fear the downvotes). .
I'm much less concerned with comments, than with posts which don't have downvotes. This means a great story with narrow focus will have a small number of votes similar to a content-less one in an area of general appeal (aka noise).
The thing I keep coming to HN for are those deep stories in things that I don't normally read about, but are written and discussed in a manner that keeps me reading and learning.
I have had that thought, being aware of some back and forth in votes. But in the end it might tend towards - I had 100 upvotes, those downvoters are just idiots. Not sure what else it really adds.