That's true for every minority everywhere - for ethnic, religious and as you mentioned sexual minorities. The threat that the majority would turn the tables on you is always a future threat.
It's a matter of degree surely. There are plenty of places in the world where a straight, white Westerner can live as a minority with relatively little fear of violence. Almost none for a gay person.
Gotta love it when HN downvotes you for reasonable comments!
To the downvoters of OP, where is safe from harassment and attacks? Here in Seattle, the queer haters literally roll in and beat us up for fun, even enclaves like Fire Island aren't going to be able to avoid this threat.