I'm sure HN wasn't a lot of things but has become them. For the majority (I'm guessing) of users, HN isn't a place you can downvote. Perhaps this experience of the website should be widened. I respect that I am on a different website, but that doesn't mean I shouldn't campaign for it to change.
I think it would be the best thing in the world if they got rid of user karma altogether, and either made it invisible or just applied it to stories. I can't imagine it happening any time soon, though.
But then again, shadowbanning being publicly reversible seemed like a major step in the right direction, and HN survived that, so I suppose never say never.