That's probably the single thing about identity politics that crosses me the worst: there are actually legitimate points in there, but they're so deeply buried under assertions like "you can't say X around me because I'm Y" and "P's have mistreated Q's for so long that Q's deserve special treatment from P's in recompense", which may not be contested in any fashion but must be taken as axiomatic, that ordinary people of all stripes, who would otherwise be quite sympathetic, instead flee screaming into the night.
I am not saying "you can't say that". I am saying that the context of who is speaking matters quite a bit. It may not seem 'fair' that when speaking on how hurtful a statement is to a particular group, whether you are actually a member of that particular group is highly relevant, but that perception of unfairness is itself an aspect of privilege.
Basically, if you make a joke about a group you aren't a member of, don't expect to be able to tell someone who is a member that it's 'just a joke', and that goes double if you're a member of the dominant group (whichever that happens to be in context).
Personally, I hit almost all of the privilege checkboxes with the (very mild) exception of religion/ethnicity. Guess what? If you aren't Jewish, and you tell me a Jewish joke, and I take offense (and I might even if it is a joke I have used myself), and I then call you out on it, 'lighten up' just isn't an appropriate response, regardless of the fact that you may not have privilege in some other area.