Almost no one claims that IQ tests are invalid, they really do estimate g, very strongly correlated with IQ. What is often being questioned is the strength of correlation between scores and life outcomes. And of course there’s the incendiary nature vs nurture debate.
Now if your test is to select which individuals from a minority group can go to college, of course not! How dare you?