Whiteness and maleness are not and cannot be material facts and it is a category error to insist otherwise, nevermind elevating them as the axiomatic center of your worldview.
The inequity resulting from the material consequences of appearing in, being raised in, and interacting with society as white and male is a well-established statistical fact.
A nice bit of wordplay but that's nonsense. The difference between statistical facts and material facts is, one is a(system-scale) summary of the other (individual-scale) measure.
I fail to see the nit you're picking at with that second bit. Are you saying they are categories? Suppositions? Made up? Do you mean to imply then that they are "spooks" and their presence in the minds of physical agents has no bearing on physical outcomes?