Yes, it's presumably a problem, but you also can't know why somebody is downvoting. They might believe a comment is dishonest or distracting or otherwise low-quality or non-curious, or breaks one of the subjective site rules.
I try not to make assumptions when I get downvoted. I think: Maybe I was a little sarcastic or aggressive or I implied something weaselly; or maybe the downvoter just disagreed with me. I don't assume either way, and I don't guess about moral high ground or whatever. All my comment's points tell me is what delta of people clicked the down arrow, not what was in their heads individually.