Is explaining the decision process a requirement? It can still solve extremely complex problems, problems humans couldn't dream of ever solving on their own (e.g. Deepfake).
Eventually yes. At some point you need to not just solve the problem but convince me that you have solved it. If I tell you I proved p!=np you wouldn't believe me, unless I can explain how I proved it so you can replicate it. (Note if p=np it is likely I can take my proof and construct an algorithm that that can only be created if it is true - thus hiding my real proof instead showing a proof by contradiction - if it wasn't my algorithm wouldn't solve this problem that is solves)
Thus if an AI say p!=np without explaining why we won't know if we believe it unless the process can be explained.
Thus if an AI say p!=np without explaining why we won't know if we believe it unless the process can be explained.