Farebox recovery is so damn low that I can only think the fares remain as an excuse to kick homeless and other undesirables off who aren’t technically breaking any rules or laws.
It's a pretty new topic in the US and studies are being conducted on this. AC Transit recently did one (I can link it if there's interest.) Indeed, farebox recovery is low enough that one of the main arguments for fare enforcement is to kick problem riders off. This is most impactful for operators who have to deal with the passengers in a way we riders do not.
I’ve always thought that issuing free passes to all appropriate taxpayers/residents would be a great way to go. Then you can raise the non pass price to soak those tourists!
They're there! AC Transit runs a program called Clipper Start. The problem is, low income folks just don't take advantage of it, and it's (without further study) unclear why. This was identified during the session on fare enforcement and exactly as you say, there's interest in promoting it more heavily and trying to jack up the price on tourists or high-income leisure riders.