[tzs]

Understanding what is going on in a neural network (or any other kind of machine learning mechanism) when it makes a decision can be important in real world applications.

For example, suppose you are a bank and you have used built a neural network to decide if credit applications should be approved. The lending laws in the US require that if you reject someone you tell them why.

Your neural network just gives a yes/no. It doesn't give a reason. What do you tell the applicant?

I have an idea how to deal with that, but I have no idea if it would satisfy the law. My approach is to run their application through multiple times, tweaking various items, until you get one that would be approved. You can then tell them it was that item that sunk them. For instance, suppose that if you raise their income by $5k, you get approval. You can tell them they were rejected for having income that is too low.

[nitrogen]

I have an idea for a company related to this concept, but for hiring and job training.