Nobody forced the banks to participate in those markets. The government interference has been limited to demanding that banks treat the public equally instead of reinforcing structural discrimination.
You don't get to take advantage of structural discrimination and then whine about how hard it is to have to treat people fairly. If that's a problem for a bank then they should find another asset class to invest in.