[grues-dinner]

Non-agentic AI is already "killing" people by some definitions. There's a post about someone being talked into suicide on the front page right now, and they are 100% going to get used for something like health insurance and benefits where avoidable death is a very possible outcome. Self-driving cars are also full of "AI" and definitely have killed people already.

Which is not to say that software hasn't killed people before (Horizon, Boeing, probably loads of industrial accidents and indirect process control failures leading to dangerous products, etc, etc). Hell, there's a suspicion that austerity is at least partly predicated on a buggy Excel spreadsheet, and with about 200k excess deaths in a decade (a decade not including Covid) in one country, even a small fraction of those being laid at the door of software is a lot of Theracs.

AI will probably often skate away from responsibility in the same way that Horizon does: by being far enough removed and with enough murky causality that they can say "well, sure, it was a bug, but them killing themselves isn't our fault"

I also find AI copilot things do not work well with embedded software. Again, people YOLOing embedded isn't new, but it might be about to get worse.