The supreme court agrees that affirmative action is constitutional (within certain bounds), see the recent Fisher v. UT case, or Bakke for the original example.
I'm not saying it's not constitutional, just that it is unrelated to civil liberties and should probably be outside their purview if they're to be a politically neutral organization.