Comment scores do affect karma, but the amount is softcapped. The more downvotes you get, the less each individual downvote counts towards your karma. It doesn't kick in until you're past a handful of downvotes, so a score of -5 will still drop your karma by 5, but a score of -50 won't drop your karma by the full 50.
If a comment is controversial (i.e. it gets both a lot of upvotes and a lot of downvotes) but still negative (e.g. you got 300 downvotes and 200 upvotes), you can even end up gaining karma overall because upvotes aren't softcapped the same way.
There are two types of Karma, post and reply. However, both up and downvotes seem to have diminishing returns on your account totals even if they show up as the full amount on the post. I know I've had a few posts that got 300+ points but only impacted my account total by like 100 or 150.