Almost universally prosperity is gained through privilege, compounded over generations. Privilege being rules/customs/systems that favored your group over others.
To be fair, the traits OP mentioned are heritable, and so to a large extent come from the privilege of having [tall | intelligence | athletic | beautiful] parents. So privilege doesn't explain why you'd feel guilty about one and not about the rest.
If one was aware that the prosperity disparity is predominantly due to circumstance and if one had empathy for other fellow human beings. Then I think it would be a natural response to want to alleviate other's suffering by sharing some of their prosperity. You can call it guilt, or perhaps just empathy. The other heritable attributes can't be easily be shared.