As the link points out, the real number of defensive gun uses is 1/2 to 5 million per year, which strengthens your argument. The 674 are just those recorded by law enforcement.
I think the main origin of the huge discrepancy is the definition of the “use” of a gun in self defense. The vast majority of these uses do not involve actually discharging the firearm, but merely indicating to an aspiring assailant that you are armed. A mugger pulls a knife, you pull aside your jacket to reveal the holstered pistol, he runs away. You do not inform the police (people stopped bothering in big cities decades ago), and it never becomes part of an official statistic.