[vmarsy]

Even if you are careful to remove all features you think could tell the race (let's say geographic location etc), it could still be giving harsher sentences to minorities if those are treated differently upfront. For instance, it would make sense for the model to learn that the more charges count the harsher the sentence. This sounds right: a bank robbery should be less punished than a bank robbery + a carjacking, at least in the US judicial system.

But now let's say minorities gets a "resiting arrest" charge on top of their original charge more often, because of many factors such as the police bias, the bias of the minorities towards police, etc. (By bias here I mean all spectrum : racism, but also fear of the police etc).

As long as minorities are treated differently upfront, then the model would treat them differently.

[mproy]

But if all of that is true, the model doesn't change the equation. The problems you outline need to be addressed upstream. Trying to account for and correct them at the point of sentencing is very problematic.