If they were of the same race as your dad, the question "how come these people that aren't even from here have a better life than me?" would still be valid.
If he can tell they're not from there, then he'll be mad. If he thinks they are fellow Englishmen he'll be less mad. That's the point, how prejudice often has a ground in economic conditions