Colds are pretty common, influenza not that much. In my 39 years of age I typically catch a cold almost every year, but the flu only once in my entire life.
I read once that many cases of the flu are really food poisoning. I got H1N1 and felt like I was dying for a week. Looking back I now think it's the only time I've ever really had the flu.
Influenza can be mild, even asymptomatic, so unless you are really able to test; you probably don't have good numbers for how often you've caught the flu.
Influenza is rare? Maybe rare for you if you habitually get the flu shot twice a year, but even then the efficacy of the flu shot can be as low as 50%. Influenza is very common, most people have experienced it.
https://www.bbc.com/news/health-31698038