[intended]

Yes, but Humans can go to jail.

You assume that your opponent is reality.

your opponent is other human beings. It takes precious little for a motivated person to learn how to hide malfeasance behind a black box.

Faith in un-corrupted black boxes should be considered the same way as faith in the un-coruptible internet

[Houshalter]

No one has any incentive to corrupt the black box. There are many, many cases where systems have terrible incentives that ruin everything. But I don't see how this is one of them. E.g. the bank has every incentive to make it's loan system as accurate as possible.