Accurate or not, the perception of someone being a Nazi made them punchable. It's not hard to argue people desiring to punch Nazis are probably not anti-Semetic.
Of course it's hard to argue that. Nazis are the great bogeymen right now, wanting to punch them might have little to do with feelings about Jews, and everything to do with just looking for an outlet to attack "the bad people" however defined.
When we read historical cases of witch trials or executing "demons" or "possessed people", it's the same thing.
This goes back to the oppressors and the oppressed. In the case of Punching Nazis, the way I read it is the meme of the anti-fascist that punched Richard Spencer during an ABC interview in 2017[1][2]. The post-modernis in me also like to point out that before then Punching Nazis was endorsed by Steven Spielberg when he directed Harrison Ford to do that as Indiana Jones[3]. And—of course—the supreme glorifies of violence against Nazis Quentin Tarantino who doesn’t let a movie go by unless a Nazi, a rapist, KKK members, etc. get severely tortured, bombed, burned with flamethrowers, etc.
Back to 2017, Richard Spencer is an actually nazi. He is a white supremacist that routinely spouts hate speech against Jewish people. In the case of the ABC interview the oppressed were Jewish people, and the oppressor was Richard Spencer. The Anti-facist very much cared about the Jewish people when he punched Richard Spencer. If Richard Spencer weren’t an oppressor of Jewish People, he wouldn’t have gotten punched.
When we read historical cases of witch trials or executing "demons" or "possessed people", it's the same thing.