being labelled neonazi is about as easy as being labelled "libtard". The very left oriented people often have a strong sense of moral superiority, and will directly accuse anyone who does not agree of being a neonazi. in general i would say though, when you are pointing with one finger, 4 are pointing back at yourself (and by this I dont mean you, its an expression)
Except that when you're marching with swastikas, and displaying nazi signs and slogans, it's kind of difficult to claim you're not a nazi. But you keep right on assuming you have a moral position.
I think that illustrates the way these things work pretty clearly.