[frabert]

I think it could be easily attributed to the fact that most non-native speakers learn English at school by studying its grammar in written form, where the two words are distinct. Native speakers, instead, learn English as their spoken language, where the words sound basically identical to each other.

[notRobot]

Yes, true. "have" and "of" don't sound same at all in almost all non-native accents which helps.