Not necessarily, if hypothetically silver people make up 20% of the population, but due to cultural bias in testing they make up 10% of the student body you could correct for that bias without discriminating.
For an actual example, rich people tend to look better on Harvard applications, but preform worse at Harvard than poor people. Adjusting for a bias in the application process would not be discrimination based on income.
If you strongly feel that affirmative action is wrong then the best way to end it is to end racial segregation in housing and K-12 education. When colleges see that all kids are getting the same opportunities they will end their programs to correct for that societal flaw. It seems very shortsighted to think that displacing affirmative action students is the best way to improve our society or that not getting into Harvard will take something from you that you deserve or need.
For an actual example, rich people tend to look better on Harvard applications, but preform worse at Harvard than poor people. Adjusting for a bias in the application process would not be discrimination based on income.