This is categorically false. People who love and want to defend free speech go there. If you want to continue living in a bubble world, stay on Twitter.
It doesn't have to be a 'Nazi symbol' for self professed Nazis and white nationalists to identify with it. Check out the mentions of any prominent Jewish person on twitter, it's a cesspool of Pepe frogs telling them to get in ovens or whatever other idiocy they've come up with today.
Case in point, David Duke saying "you can't zog the frog":
Does Twitter have no right to freedom of speech, when there are alternatives to it? If they don't want to be associated with white nationalists, then why should they allow themselves to be?
If the difference between Twitter's speech limits and Gab's is that the latter allows Nazis and harassment, I think you'll find most prefer "less" free speech.