There is no downside to society if banks are less profitable. There is a definite downside to the banks, though. And given the chance, at least some banks will try to get out of that downside.
if banking was less profitable, all else being equal, you'd expect that there'd be less banking activities including loans. Less loans means less economic activity from what those loans are funding (such as business loans, or construction loans etc).
Why would you expect that? It doesn’t seem to follow at all… If banking was less profitable, they’d more likely want to try and increase volume (issuing more loans) to try to make up for it and make more profit.
(Which is why bank regulation is incredibly important, because without strong regulation banks often do start to chase profit by lending to more and more risky customers, and when that unwinds you get what happened in 2008)