Hong Kong is a city state however, so it is probably true that most of their residents travel abroad each year (if we count China as traveling abroad, but even if we don't, > 50% is probably about right).
I assume you know that the majority of Hong Kong citizens live in poverty. They can barely afford a roof over their heads, much less international travel.
But perhaps you've never experienced poverty, so you may not know.
1.6 million out of 7.3 million people is definitely serious, but hardly a majority. I've been to the SAR a few times, and don't think it is especially poor, especially when compared to even China's richest city like Shanghai.
As for my own personal experiences, please don't make assumptions like that. You have no idea what I've been through in my life, and it is rather not relevant to this discussion.
Perhaps I overstated the poverty in HK. But my point was that many people there cannot afford to travel. They live paycheck to paycheck; they have very little in the way of savings. I know people in HK, including family. Most people in HK are not well-off.
When you add these to the people who are living in poverty, that doesn't leave for a lot of travelers.
The ability to afford an apartment (expensive, even rent) and the ability to take a holiday to Thailand are completely different costs. An HK resident could be homeless (unable to afford rent) and still easily have the ability to take a holiday to Phuket. That's how crazy HK is, at least.
But perhaps you've never experienced poverty, so you may not know.