[version_five]

Two separate things: first, you're correct, there are lots of attributes that can be accurately inferred from appearance (for varying accuracies). I don't see a point in pretending otherwise.

Second, unless demonstrated otherwise, most determinations you could make, e.g. wealth, are not causal, they are effectively a computerized stereotype that looks for some common features the majority of each class share. To me this means facial features are not a suitable basis for a decision anywhere you wouldn't feel comfortable stereotyping.

Put another way, you can easily propose rules that are correct on average but horribly unfair to those that don't conform to the rule. This is true for ML as it is for any other rules. The only ML specific thing here is maybe the basis for the predictions gets obscured to some as something deeper than it is.