[sumitviii]

I believe this is not the correct thread, but how can anyone sift through so much data, in general? Private companies need simpler things, like people you are likely to know in the real world, from the data they acquire. But intelligence agencies need actionable intelligence. That would require something way more intelligent than a simple spam filter.

[wongarsu]

That depends on the data you are talking about. The operations described here don't seems to collect huge amounts of data. If you're talking about the usual dragnet surveillance: a lot of it seems to be relatively simple filters and simple data correlation.

For example, you can build huge social graphs with simple metadata. Then you can search for all people who communicate a lot with people who communicate a lot with some known terrorist leader. Of those people, you take just those using tor. If any of them plans to enter the US, you flag them to be detained and searched at the airport. If any of them is already in the US, you can tell FBI to check them out.

Or you can look for sudden changes in message volumes. If terrorist leader A suddenly starts to communicate a lot more with random person B and random person C, who in term start communicating with other people, you suddenly have a whole list of people who might be planning a terrorist operation.

Of course you still need huge computing capacity even for these relatively simple operations, but they certainly have the funds for a few datacenters.

[lbotos]

I think that's the goal of firms like Palantir. To help filter that volume of data.

[scott_karana]

They don't necessarily need actionable intelligence: just supportive evidence to back up recon/threats/blackmail as part of other agency's actions.

[Estragon]

  > intelligence agencies need actionable intelligence

For the most part that hasn't really been how it's worked so far. Generally intelligence agencies have used the information they've gathered so far to try to manipulate people.