I don't think that's accurate - I want points back because it allows me to see what other people have voted. Because I trust this community I'll be fairly certain these ar ethe comments that will hold the most value for me.
I trust my judgment better with the lack of quantified information on how the group as a whole has behaved. Perhaps I'm just projecting but I trust the judgment of others better under the same circumstances.