Not entirely. Humans are able to use well described and backed logic in decision making. Ever seen AI write out its decision logic, in a form that's portable to other AI?
People past few years old can output at least partial rationale for behavior or decisions. Systems like BERT are at best comparable to a pre-linguistic 2 year old.
I admit that it's unfortunate that AI can't write out their decision logic but I would argue that is because there hasn't been enough resources put into explainable AI. Considering the increasing use of these algorithms, I don't know if that is even a high priority.
I tend to think that people are not as logical as they like to think they are, myself included. Not to say there isn't good reasoning, just that much of our decision making is emotional and habitual over some pure sense of logic.
Systems like BERT seem perfectly rational to me. Are they not just following a set of rules on a given input to modify a state?(In the most simplistic sense of computation). I think the confusion is more over what the goal of these programs are and how do we encode that. This reminds me of the ai system that would pause the game of tetris so that it could never lose. Not we it's programmers intended but still accomplished it's "goal".
While people are not as logical as we like to be, we all are logical. I can teach someone the rules of math - my method of teaching might be (probably is) bad, but if the student tries he will learn those rules. Latter on when given a test the student can show his work and it will be much the same as every other student trying to explain his reasoning: operations like "complete the squares" have been well described and reasoned out.
Likewise chess masters can explain their thought process while looking at the next move and other chess masters will agree the lines of thought are good (they will probably ask why not some other equally good line...). We know this explanation is good because students can watch the experts explain their thought process and replicate it in games to a small extent and to better.
Humans make decisions based on arbitrary and capricious preferences, and are good at coming up with logical-sounding reasons to justify those decisions (eg: rejecting a candidate because she is a black woman, but saying it was because she didn't communicate well). A neural net makes a decision based on a perfectly reproducible sequence of calculations. There has been a lot of work published about how we can explain the output of neural nets, and IMO this has much better prospects than trying to explain a human’s decisions by scanning the brain or whatnot.
Is explaining the decision process a requirement? It can still solve extremely complex problems, problems humans couldn't dream of ever solving on their own (e.g. Deepfake).
Eventually yes. At some point you need to not just solve the problem but convince me that you have solved it. If I tell you I proved p!=np you wouldn't believe me, unless I can explain how I proved it so you can replicate it. (Note if p=np it is likely I can take my proof and construct an algorithm that that can only be created if it is true - thus hiding my real proof instead showing a proof by contradiction - if it wasn't my algorithm wouldn't solve this problem that is solves)
Thus if an AI say p!=np without explaining why we won't know if we believe it unless the process can be explained.
People past few years old can output at least partial rationale for behavior or decisions. Systems like BERT are at best comparable to a pre-linguistic 2 year old.