Since mathematical skills (broadly defined) are a component of IQ, this is dangerously close to "Basketball is a getting the ball in the hoop game. Anyone who tells you otherwise is kidding themselves." True, but less informative than you might think...
(This is not to comment on the underlying question of whether everyone has the potential to be a great mathematician, just a particular type of reasoning).
Because they are calibrated not to measure innate intelligence, but our own cultural norms. The canonical example might be, imagine you were lost in the Australian outback. Would you rather have Albert Einstein as your guide, or a skilled Aborigine? The Aborigine would likely get a low score on logic puzzles, but at highly skilled activities like way-finding and tracking game, he would beat Einstein hands down.
IQ doesn't measure intelligence, it measures artifacts of intelligence that appear in our culture.
Also, it's a mistake to think that IQ is static. There have been studies that show wild swings in IQ over time, especially in teenagers, and even changes in the collective IQ of a culture (the Flynn effect).
In addition, it seems likely that a person could study and improve their IQ score.
The counter-argument is that although IQ tests have flaws, they are still to some degree a predictor of success in life.
I think another great example would be IQ tests in the army during WWI (and I think II?) in which one of the questions asked about a type of gun, which no one today would know, and other questions were like that as well. The point is that it measures cultural perceptions of intelligence, ie during WWI knowing your guns was considered intelligence.
Mostly in the same way the basket ball is a tall persons game. Yet there have been a lot of very good basketball and quite a few brilliant players below 5'10" (the US median height).
(This is not to comment on the underlying question of whether everyone has the potential to be a great mathematician, just a particular type of reasoning).