I don't think anyone is advocating for this, and it's not limited to AI (software with a bunch of rules has the same problem). The EU went so far as enshrining into law the right to always escalate any decision to a person.
But we DO advocate to have everything encrypted with no possibility to recover it if you lose access to the master key. That creates interesting ramifications in OP argument.