[feoren]

(Arguing with the point you present, not against you, since it seems like you mostly disagree with it too.)

Reason 1 to reject spying for this reason: literally any spying could be justified this way. Every butterfly's wing flap could theoretically contribute to a "national security risk". In line with that great quote that "If no amount of risk is acceptable, then any amount of surveillance is justified." There's no way to use this logic to ever argue that you shouldn't spy on someone.

Reason 2 to reject this: when we start broadening the scope of "national security risk" to include things like "economic disruption", then essentially anything that's outside of the status quo is suddenly classified as a risk. Basically our spy agencies' jobs become protecting the entrenched interests of those already in power. A technological breakthrough in battery tech has a big potential for "economic disruption"; is that a national security risk? What if it's invented in Iran? Would the U.S. sabotage the (hypothetical) "Iranian national energy laboratory" if it were on the cusp of a revolutionary technology that could weaken the U.S.'s economic position?

Reason 3 to reject this: we get it wrong. If the CIA's "terrorist detector" says Joe Schmoe has a 99% chance of becoming an active terrorist, based on his religion, political views, age, recent Google searches, facial hair, income, and gait, what is the actual probability of him becoming an active terrorist? The standard "Bayesian false-positive" puzzle applies here: the base-rate of humans committing actual terrorist acts is extremely low. Even if the CIA's terrorist-detector model is 10x better than OpenAI's wet dreams, there's no way it's more than 99% accurate. With a base rate of something like 10^-6, we're looking at a roughly 0.01% chance that Joe is actually a terrorist, given a very accurate model giving a 99% positive result. Do we think the person making the decision about whether to make Joe disappear is aware of these subtle nuances in Bayesian statistics? Even if he did, he also knows that if Joe did go on to bomb an office building, there might be a headline soon that "the CIA's own model said Joe Schmoe had a 99% chance of being a terrorist and DID NOTHING!!" So of course Joe disappears forever, based on a futurecrime, because he Googled the wrong thing at the wrong time in his life with the wrong skin color and facial hair.

Put all these reasons together and you can get some really questionable behavior from these agencies, and that's even assuming it's all done with the legitimate good intention of protecting the U.S.

[schoen]

> There's no way to use this logic to ever argue that you shouldn't spy on someone.

This reminds me of a talk by Halvor Flake in which he says that people in intelligence agencies are incentivized to try to know everything, in order to minimize the chance of being surprised by anything.

> A technological breakthrough in battery tech has a big potential for "economic disruption"; is that a national security risk?

I think it's kind of an abuse of language to call it that, but I wouldn't be surprised if many people intuitively thought that yes, this should count as a national security risk.

> The standard "Bayesian false-positive" puzzle applies here: the base-rate of humans committing actual terrorist acts is extremely low. [...] Do we think the person making the decision about whether to make Joe disappear is aware of these subtle nuances in Bayesian statistics?

I think that people at, say, the CIA are somewhat aware of them because they have a notion of confidence levels of assessments, and they're also used to the idea that they have a lot of noise to deal with. Like when they were trying to find Osama bin Laden's house, they presumably had many many many many ideas of places that could potentially be his house for some reason, but then they actively tried to disprove those hypotheses (like scientists!).

They will also literally say that they assess certain things with low confidence. I don't know if some of the analysts writing that maybe actually have an explicit Bayesian probability and are mapping that into particular ranges...?

On the other hand, I think that military personnel in, say, the initial U.S. occupation of Afghanistan did not have this perspective and were happy to detain people for minimal and highly speculative reasons. (Also, I guess the base rate of people in Afghanistan during the war who were violently opposed to the U.S. was much higher than the usual base rate worldwide.) They got a number of tragic false positives which led to enormous injustices.

I think part of what you're getting at here is that governments' power when acting to actually influence the world is, well, a matter of life and death. So things like the base rate fallacy are just as big a deal there as they are in cancer treatment, or, if you believe there's a relevant moral asymmetry between killing and letting die, an even bigger deal. Also, lots of kinds of agents of lots of governments are rarely ever very accountable for their decisions or decisionmaking processes.

I think this is kind of an argument against surveillance and espionage (you can think of the scene in Good Will Hunting where the main character says he doesn't want to work for a spy agency because he can vividly imagine a false positive scenario in which a foreigner gets killed because of his work), but I think overall it's much more an argument for more transparency and accountability in uses of force. Like, you don't want to give lots of information to the CIA if they're violent maniacs because they might use that information recklessly. But while giving them that information might also be highly questionable on moral and legal and political grounds, it seems like the biggest problem in the scenario is if the CIA is composed of violent maniacs, or if it's incentivized to cultivate rather than restrain people's tendencies to violent maniacal behavior.

As an analogy about the base rate argument, people have argued that it could be beneficial for people's health to perform fewer tests, in cases where false positives often lead to wasteful or dangerous medical interventions. But if there's nothing bad about the tests themselves, theoretically it would make more sense to give doctors more information rather than less, while trying to improve their decisionmaking.

[feoren]

Great points in general, but two things:

> you don't want to give lots of information to the CIA if they're violent maniacs because they might use that information recklessly ... the biggest problem in the scenario is if the CIA is composed of violent maniacs, or if it's incentivized to cultivate rather than restrain people's tendencies to violent maniacal behavior.

The problem is that the system can act like a violent maniac without any individual human being one. Each person running a model, typing into Excel, making just a small decision, can add up to enormous injustices. Similar to how an experiment that just tries to collect all the data can be much more vulnerable to p-hacking and researcher bias, even with well-intentioned researchers who would never dream of intentionally faking or fabricating data.

> As an analogy about the base rate argument, people have argued that it could be beneficial for people's health to perform fewer tests

And I think a similar argument could be made to show that, in the face of flawed models (possibly much more flawed than diagnostic tests), it may be beneficial for the country if spy agencies actually had less information. I think this is why I included "gait" as one of my features that identified my theoretical terrorist. Maybe it's a useful feature, maybe not. Maybe he injured his foot yesterday, causing him to walk like someone who's planning a bombing. To make it really obvious, how would you feel if you found out the CIA was using Zodiac signs as one of the features of its terrorist-detector model? I'm sure it's not literally doing that, but I'm less sure it's not using features that are just as spurious.

[schoen]

I think this is all fair, but it's really so hard to know.

Even this idealized scenario about the terrorism-detector and taking action based on it... well, it's one that came up a lot around the post-9/11 environment (with selectees and the no-fly list [the rare case where people could actually experience the government making an adverse determination against them on classified grounds and get even the one bit of information] and the Guantanamo detainees and the drone kill lists).

However, the task of deciding if person X is a good guy or a bad guy, or trying to make a list of the bad guys within some population, is a very small part of what the intelligence community does. And that's partly because of the super-expansive notion of national security that we've been talking about, where so many parts of the government feel that it's their business to know everything that's going on in the world, largely without even thinking about the "who is a terrorist" question at all. As we discussed elsewhere in the thread, the government does not think that Petrobras is a terrorist. But somehow it thinks that it should know what Petrobras is going to do in the future, including "by hook or by crook".

I guess there are really at least two threads in the conversation, one of them being the inherent morality of surveillance and espionage activities, and another being "given how (one might think, but probably doesn't actually know) a government makes choices about the use of force, is it a good idea for that government to have access to more and more information?".

I think one of riversflow's points in reply to you was basically that, if a government doesn't make choices about the use of force in a good way, that's already a problem, which is not clearly improved by reducing the government's access to information, although you're also right that it could be. (And I've seen Bayesian arguments about that before, including a Bayesian justification for some legal standards involving in the rights of criminal defendants.) But overall, if we know so little about how those decisions are made at all, it will be hard to be very confident about whether more or less information would be likely to make them better or worse!