[grzm]

Can you provide links to the comments where people are having problems downvoting if they have 501 karma or more?

There is no other criteria for downvoting that I know of. When I achieved the threshold, the downvote button appeared. I didn't have to apply for anything, no one emailed me and said "We think you're swell! You get to downvote!"

I get the impression that you think there's some sort of secret down voting group. I haven't seen any evidence of that, either in behavior, or mentioned in comments.

It's fine if you don't want to downvote. There's no requirement to do so, and many people don't.