[ummwhat]

There's no consistent grammatical rule. Certain words and phrases are obvious warning signs about the speaker, but you have to be somewhat immersed in the culture to notice. Usually the literal meaning of the word / phrase really is an innocent neutral descriptor.

No one ever starts a sentence with "The Jews control" and ends it with "the home owners association on our block and have done a phenomenal job managing it."

No one ever says "The Japs" followed by "have really contributed to our culture in the form of anime".

No one ever says "that n***" followed by "is an inspiration to us all and we are proud to have him in our community."

No one ever uses the word "tranny" followed by something well thought out and respectful of transexual people.

None of these words/phrases have an offensive dictionary meaning, but they are all rightfully associated with offensive speech.