That's empirically false. "Racism" is whatever speech will convince enough people to call you "racist", and one person in twenty is more than enough to do it.
Whats empirically false changes person to person and culture to culture as demonstrated by your arbitrary definition of racism. I do not define Racism as actions other find racist.
I'm not interested in a debate about the meaning of truth - replace empirical correctness with everyday practicality and my point stands. You're free to define the term "racism" however you want to. However, you might regret ignoring your personal and cultural context if you care about not getting called racist.