[dragonwriter]

> Before that happens, the whole world would need to be under continual observation.

I don't see any reason that is a necessary precondition for an evolving reliance on AI judgement. Though obviously there are potential interactions between surveillance ubiquity and that.

> Then the AI wouldn't be saying, "I think it's likely you did this", it would be saying, "Here's the video of you doing this".

Crimes tend to have elements that cannot be shown on a video, but which might be deemed likely based on a video, like knowledge and intent.

So, it will exactly be “I think oy is likely that you did this”.

[BizarroLand]

I don't know how I would feel about allowing AI to put humans in prison without incontrovertible proof.

Any black box thing, (that is, any system where raw data comes in and a result comes out without everyone involved fully understanding the process) should not have the authority to take a human's freedom from them of its own volition.

I would much rather utilize them as a Sherlock whose job is to assemble the evidence that is available and state their conclusions to the jury, who could then make their decision based off of the evidence.