Sexism and racism....are huge areas of concern...whereas tall men and good looking people are treated much better than short men and ugly people...our morality, in the West, is inconsistent and arbitary..
There's a limited amount of energy and time available, and fixing problems is difficult and uncertain. If we wanted something more consistent, we would probably have to pull a Peter Singer and donate the bulk of our resources to the most unfortunate. I don't think we really want a less arbitrary system even if we could define it!
That would open up a Pandora's box, honestly. Even if the goal were 20% consistency, short or ugly people would definitely not make the cut compared to helping people in the third world or other potential targets of redistributive justice. Our entire economic and social edifice is built on certain other countries being poorer and less energy intensive.
Now if the goal is simply to be a bit more mindful, that would be doable i.e. avoiding unnecessary remarks about height the way we avoid saying hurtful comments in other contexts