What do you call people that hang out with Nazis? It sounds like you are nitpicking that adjacent white supremacist groups with their own iconography (and merch) are somehow different.
That's sort of the issue with informal groups and protests. Anybody can show up, and they may or may not be rejected by the crowd at large. When the loonies are kicked out, it doesn't make the news. A left-leaning example of this is when Boogaloo Boys participate in Black Lives Matter protests. I'm not saying they're equally common, but the concept is the same.
I don't think anyone believes that these groups were a statistically significant component of the protests. Of course, if we legitimately think that Nazis and white-supremacist groups present a serious problem in Canada we should be addressing it - not using it as a kind of guilt by proximity argument to delegitimize protests of other causes.