I visited HK in 2018 for a conference. All I can say is that I am glad that I went with my white friend, and speak English all the time. Otherwise, I would be afraid of walking on streets.
That is completely nonsense. There were 50 millions[1] tourists from China to Hong Kong via Individual Visit Scheme in 2018. If it is that dangerous, how come so many Chinese tourists keep visiting Hong Kong?
I wasn't trying to say it is a shared experience. A few Mandarin speakers were attacked in streets prior to my visit. It definitely had me question my safety in the city. I didn't have any issue while I was in HK, but it was just how I felt, I wouldn't feel safe without my white colleague.
[1]: https://www.tourism.gov.hk/en/tourism-statistics-2018.php