Empathy goes both ways. You can recognize them being unfair while still appreciating their reasons for being unfair.
People seem to have this notion that there's some theoretical possible world where everything is completely moral, and we're just failing to get there. But that is not true. You get locally moral and globally moral arrangements, and they're not necessarily going to mesh. It's just like any other large system.
Guy can be justified from their perspective, people can be justified for distancing themselves from him. That's life. Having a reason for something is further the bare minimum, not the endgame.