[khafra]

Extrapolate the result of adopting this AI to 2030: The unemployment review AI has been in place for years, and the human staff has been reduced to one or two inexpensive recent graduates trying to work their way up to a real position; they don't have time for more than rubber-stamping the system's decision.

Imagine there's some kind of problem. Say, an enterprising journalist shows that the system has somehow broken; 1/3 of its decisions are essentially random, rendering thousands of deserving workers homeless. What does Nevada do? Everyone who knew how to do reviews manually has gone on to other work or retirement.

That institutional knowledge isn't coming back. Before you can turn the AI off, you need to build a whole new human-driven review process from whatever legible information you have archived somewhere.

For a single process which is catastrophic only for improperly rejected individuals, this process is expensive but doable. But unemployment review is not the end; AI is not going to stop at doing the vast majority of the decision-making for only one single, non-vital process. Add in a few more--say, 911 dispatching, payroll processing, a few utilities--and putting them back under the control of competent humans will quickly grow beyond the capacity of a the state.