"Win" the argument as in "stop the conversation because when someone drops that nuke they have made it clear that there's no rebuttal that will be accepted".
Luckily, once you're in the voting booth, you don't need to argue with someone to vote: all you need to do is make your marks and hand in the paper.
To be clear, there are racists, and there are valid accusations of racism. I grew up and lived in the American Southeast for a long time. I'm WELL aware of the ongoing existence of and trouble with racists.
While my personal experience is affected by being in the SF Bay Area for the past couple of decades, it is my experience that over those decades the overwhelming majority of accusations of racism in ordinary conversation are not invoked because of racism, but are used as a shorthand for "Fuck you, I'm done with this conversation and done with YOU.".