One example comes to mind, where an Apache helicopter gunner killed a journalist holding a large camera, thinking the silhouette looked like a rocket launcher.
One soldier made this mistake. What if a faulty image fitting/target classification algorithm deployed in a fleet of autonomous drones was vulnerable to a similar misclassification and started flagging some innocuous objects in people's hands as weapons? Would the rules of engagement be permissive enough for the drones to begin killing those people on their own, would there be human operators in the loop confirming and authorizing the strike?
One soldier made this mistake. What if a faulty image fitting/target classification algorithm deployed in a fleet of autonomous drones was vulnerable to a similar misclassification and started flagging some innocuous objects in people's hands as weapons? Would the rules of engagement be permissive enough for the drones to begin killing those people on their own, would there be human operators in the loop confirming and authorizing the strike?