| > There's no way to use this logic to ever argue that you shouldn't spy on someone. This reminds me of a talk by Halvor Flake in which he says that people in intelligence agencies are incentivized to try to know everything, in order to minimize the chance of being surprised by anything. > A technological breakthrough in battery tech has a big potential for "economic disruption"; is that a national security risk? I think it's kind of an abuse of language to call it that, but I wouldn't be surprised if many people intuitively thought that yes, this should count as a national security risk. > The standard "Bayesian false-positive" puzzle applies here: the base-rate of humans committing actual terrorist acts is extremely low. [...] Do we think the person making the decision about whether to make Joe disappear is aware of these subtle nuances in Bayesian statistics? I think that people at, say, the CIA are somewhat aware of them because they have a notion of confidence levels of assessments, and they're also used to the idea that they have a lot of noise to deal with. Like when they were trying to find Osama bin Laden's house, they presumably had many many many many ideas of places that could potentially be his house for some reason, but then they actively tried to disprove those hypotheses (like scientists!). They will also literally say that they assess certain things with low confidence. I don't know if some of the analysts writing that maybe actually have an explicit Bayesian probability and are mapping that into particular ranges...? On the other hand, I think that military personnel in, say, the initial U.S. occupation of Afghanistan did not have this perspective and were happy to detain people for minimal and highly speculative reasons. (Also, I guess the base rate of people in Afghanistan during the war who were violently opposed to the U.S. was much higher than the usual base rate worldwide.) They got a number of tragic false positives which led to enormous injustices. I think part of what you're getting at here is that governments' power when acting to actually influence the world is, well, a matter of life and death. So things like the base rate fallacy are just as big a deal there as they are in cancer treatment, or, if you believe there's a relevant moral asymmetry between killing and letting die, an even bigger deal. Also, lots of kinds of agents of lots of governments are rarely ever very accountable for their decisions or decisionmaking processes. I think this is kind of an argument against surveillance and espionage (you can think of the scene in Good Will Hunting where the main character says he doesn't want to work for a spy agency because he can vividly imagine a false positive scenario in which a foreigner gets killed because of his work), but I think overall it's much more an argument for more transparency and accountability in uses of force. Like, you don't want to give lots of information to the CIA if they're violent maniacs because they might use that information recklessly. But while giving them that information might also be highly questionable on moral and legal and political grounds, it seems like the biggest problem in the scenario is if the CIA is composed of violent maniacs, or if it's incentivized to cultivate rather than restrain people's tendencies to violent maniacal behavior. As an analogy about the base rate argument, people have argued that it could be beneficial for people's health to perform fewer tests, in cases where false positives often lead to wasteful or dangerous medical interventions. But if there's nothing bad about the tests themselves, theoretically it would make more sense to give doctors more information rather than less, while trying to improve their decisionmaking. |
> you don't want to give lots of information to the CIA if they're violent maniacs because they might use that information recklessly ... the biggest problem in the scenario is if the CIA is composed of violent maniacs, or if it's incentivized to cultivate rather than restrain people's tendencies to violent maniacal behavior.
The problem is that the system can act like a violent maniac without any individual human being one. Each person running a model, typing into Excel, making just a small decision, can add up to enormous injustices. Similar to how an experiment that just tries to collect all the data can be much more vulnerable to p-hacking and researcher bias, even with well-intentioned researchers who would never dream of intentionally faking or fabricating data.
> As an analogy about the base rate argument, people have argued that it could be beneficial for people's health to perform fewer tests
And I think a similar argument could be made to show that, in the face of flawed models (possibly much more flawed than diagnostic tests), it may be beneficial for the country if spy agencies actually had less information. I think this is why I included "gait" as one of my features that identified my theoretical terrorist. Maybe it's a useful feature, maybe not. Maybe he injured his foot yesterday, causing him to walk like someone who's planning a bombing. To make it really obvious, how would you feel if you found out the CIA was using Zodiac signs as one of the features of its terrorist-detector model? I'm sure it's not literally doing that, but I'm less sure it's not using features that are just as spurious.