><i>How likely is the NSA PRISM program to catch a terrorist?</i><p>A better question, which involves historical understanding about the issue rather than going directly to the maths and taking the framing of the thing as granted:<p>How likely is the NSA PRISM program even cares about catching terrorists?<p>Sure, they wouldn't say no if it did. It would even justify the program to the eyes of the public.<p>But, thing is, similar programs have been going on for centuries in all governments. They care only about two things: spying on third countries (enemies, competitors, etc, things that effect commerce and military) and spying on their own citizens (population control, etc). Those programs have roots aeons before 9/11, and aeons before "terrorism" was any real concern of the government.
This shouldn't even be a question. Are we going to ask how likely is that torture is going to work, too?<p>We shouldn't allow broad surveillance of everything you ever do or say online, and we shouldn't accept torture as a means of interrogation either. This is simply deciding about the kind of society we want to be.<p>Terrorists can be dealt with in other ways, too, and one of them would be trying to avoid creating blowback and radicalizing future generations of terrorists because of current actions.
An interesting application of Bayesian reasoning to the problem of screening for low-base-rate phenomena. (And terrorist criminal activity is a lower-base-rate phenomenon in the United States, so far, than prostate cancer or other dangers that are screened for.) The mathematics, of course, is exquisitely sensitive to exactly how sensitive and specific a screening program is. (By the way, so far news reports are saying that the NSA program that is all over the front page of Hacker News, deservedly so in my opinion, is not a screening program but a data collection program, with analysis of the collected data triggered only by other kinds of law enforcement evidence-gathering. We'll see what further reporting says about that issue.)
Many of us with some (Bayesian) statistics/game-theory backgrounds are familiar with these kinds of puzzles. There are many (many) such anecdotes littered through math textbooks (HIV +/- testing comes to mind) as well as some real-life scenarios (UC Berkeley hiring practices and women comes to mind -- 1970s case).<p>HOWEVER, let me just say that two metrics in this little puzzle are simply wrong (or at the very least unfair):<p>P(+ | bad guy) is simply a LOT larger than 0.99 if you want to play the game correctly. The + comes from a POSITIVE outcome -- that is, a terrorist attack is thwarted and someone is thrown in jail. Some noise in the system (i.e. pizza orders) does NOT skew the positives down. After all, discerning between pizza orders and terrorist activity is <i>part of the algorithmic process</i> -- the automated system may flag both cases, but the buck doesn't stop there (or anywhere close).<p>P(+ | good guy), again, suffers from the same problems as P(+ | bad guy). Unless there is evidence of 1% of the people being monitored being thrown in jail for terrorism charges, that number is a lot (lot) less than 1%.<p>There have been plenty of bogus terrorism charges (see Guantanamo) and I think there may be something here. But if we want to play this game correctly, we need to be careful. To do this simulation, we need numbers that we will simply never have access to: e.g. how many terrorist threats did PRISM avert?
This is an incorrect question. Anti-terrorism is not the only thing the NSA is interested in.<p>Better question: How likely is the NSA PRISM program to provide intelligence useful to the NSA's mission?
This is a very well known problem, but one that no politician on this planet seems to understand.<p>Of course, as others have pointed out, the problem is not the likelihood of getting a bad guy. The problem is that the means are unacceptable, no matter the goal.
Is that what it's for? I thought it'd be a big retroactive database; identify someone we want to know about now, and then look back through the last year's records to see what she's been doing.
I wonder if the intense scrutiny these programs are coming under increases the likelihood of catching the next attempted terrorist attack by using them. (if you catch my cynical drift)
So? 1/10,100 isn't bad at all. Suppose you have many independent tests, each creating watchlists. Suddenly you have the intersection of individuals hitting each of these, and the number of "interesting" individuals narrows. This is not a univariate problem; they're likely correlating everything to everything, and have adaptive snooping methods based on how high you are ranked a risk. Once you're high enough, field offices know about you and will keep a closer eye. It's really not a needle in a haystack (and as a privacy nerd, it pains me to say this, but I bet it is working).