Except that math doesn't work either. 3-4 times the morality rate is not the same as 3-4 times as many dead, for the same reasons I listed above - we've already made lots of societal adjustments to change the input number before the mortality rate of the seasonal flu gets involved.
The flu has vaccines, and anti-viral treatments. If it did not the reaction to it might be very similar to how we're being told to respond to COVID-19. These are not comparable things because the options for combating them are drastically different.