[daoxid]

> I think it's important to note that human pattern recognition is basically black-box as well.

Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

[knzhou]

> But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion.

No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

> So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

You can already do that. Just change that number in the input and see how the output changes. To that extent, even the most black box AI model is more transparent than human decision making.

[daoxid]

> You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

Well, I guess it depends on how self-aware a person is. I think the biggest danger is trying to rationally explain your decision when in fact it was based mostly on your feelings, in which case I agree that the explanation is "basically a lie". One needs to be honest when something is not based on a fact but on a feeling to prevent pointless discussions. (If I hold an opinion based on a feeling then you cannot convince me that I am wrong by giving me facts.)

> You can already do that. Just change that number in the input and see how the output changes.

Makes sense. But I guess transparent models would still be generally preferable because you can fully understand how the output is produced, whereas in black-box models you might have to ask quite a lot of questions to get a feeling for it, but even then you can't be sure that you have a full understanding of it.

[rwallace]

> even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

You punctuate the second sentence as though it were of secondary importance. But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers. I put it to you that having questions answered with plausible fabrications, is actually a significantly worse situation than not yet being able to ask the questions at all. At least in the latter situation, we know what we need to be working on.

[daoxid]

> But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers.

Hmm, too me it feels like I can explain the reasons why I came to a conclusion in many (but certainly not all) cases. You "just" need to clearly identify your feelings and emotions and separate them from your rational arguments.

Anyway, these are our own shortcomings and of course don't have to be adopted by any artificially built black-box model.

[q_andrew]

This is actually false. Early psychologists thought that humans could know everything about their own brain's processes, but observational psychology proved most of their assumptions wrong. It's called introspection, and it's almost always not indicative of a person's actual inner workings.

[jlarocco]

Expert systems from the 70s and 80s had a capability similar to this. They could explain how they reached a conclusion by reporting the rules they used to get there. The problem was that interviewing experts and coming up with a huge rules database was a ton of work and didn't scale very well.

Maybe the next direction in AI will be to bridge the gap between expert systems and black box models?

[dcposch]

> we can ask them questions in order to find out how they came to a particular conclusion.

For the most basic cognitive tasks, we typically can't.

If you show someone a picture of a cat and a dog, they can easily recognize which is which.

If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...

[daoxid]

I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.

[lonelappde]

Introspectable Higher level reasoning is irrelevant. Any introspectable higher level reasoning a person can do has already been automated for efficiency.

[daoxid]

Wait, what? Where has this been automated?