[tzs]

As long as the model is deterministic in operation, even if the process of building the model is non-deterministic, can't decisions be interpreted by rerunning the model with tweaked inputs to find what would be needed to change the outcome?

For example suppose a bank rejects a credit application and they need to tell the applicant why. Repeatedly add $1000 to the income input and check again until until the model approves, and then tell the application they were rejected for insufficient income and how much more they would need to pass.

[vermarish]

Yes, this is a counterfactual line of reasoning. It's certainly legit, and I believe it's an active field of research in ML, but at the end of the day, you're still poking around the inputs of a black-box model as opposed to examining a white-box model.