[aurelian]

Any organization with secrets worth keeping must try to detect spies and repel infiltrators.

It may reduce disastrous false negatives at the cost of an increase in false positives.

In the 1940s, the Soviet Union had hundreds of agents in the executive branch of the US government. KGB records opened in the last 20 years largely vindicate McCarthy (Edit: in that the threat was real; infiltration had happened; most of his targets were Soviet agents, Communists, or associated with Communists; and security risks had to be removed from positions in which they could do harm for the good of freedom-loving people everywhere).

[lotsofmangos]

I don't see that evidence of spies in government vindicates MCCarthy.

It might do if he had restricted himself to the threat of spies in the government, rather than instituting a campaign against communists and homosexuals throughout much of US society.

Spies in the executive does not justify Hollywood blacklists, the censoring of library books resulting in book burnings, attacks on the clergy, attacks on academics, the 10000 people forced into unemployment by the loyalty review boards or the repeated burglaries against lawyers who would represent those who were targeted.

[a3n]

Yeah, it was just a vehicle for McCarthy. It could just as easily have been the scourge of crabgrass.

[jjoonathan]

According to

http://en.wikipedia.org/wiki/Joseph_McCarthy

> historian John Earl Haynes concluded that, of 159 people identified on lists used or referenced by McCarthy, evidence was substantial that nine had aided Soviet espionage efforts.

9/159. A 94% false positive rate.

[saalweachter]

You know, the problem with lists like this is that, once you've made the list, any organization with a cover-your-ass mentality (eg, every organization) is going to treat it as sacrosanct. Regardless of how flimsy the list was to start with, no amount of evidence can ever clear someone well enough. The risk isn't "chance of being wrong * real-world consequences of being wrong", it's "chance of being wrong * (real-world consequences of being wrong + career consequences of 'letting the guy on the known-threat list slip through')". The career consequences have infinite weight, so the chance of being wrong can never be low enough.

[aurelian]

Why don't you quote the next sentence? "He suggested that a majority of those on the lists could legitimately have been considered security risks, but that a substantial minority could not."

[jjoonathan]

Because it's nearly tautological: everyone on that list got there for some reason. The fact that a majority of those reasons weren't entirely made up isn't an indication of their quality as predictors of espionage.

Since we now have access to the ground truth we can compute the actual false positive rate. The great-grandparent post suggests that such an examination would vindicate McCarthy, but I have trouble sympathizing with this view. A 94% false-positive rate (96% if you use McCarthy's original figure) is bad no matter how you slice it.

[aurelian]

McCarthy identified several agents actively engaged in espionage. KGB records identify hundreds more that he missed.

McCarthy identified many more people in sensitive positions who were security risks, many with Communist associations. Even if those people were not actively engaged in espionage, they should not have held sensitive positions.

KGB/NKVD archives reveal they had hundreds of agents in the executive branch in the 1940s. McCarthy spurred the removal of Soviet moles, Communist sympathizers, and other security risks from sensitive positions. In this, he did the US a great service, as regrettable as false accusations are.

Edit: In response to your questions, it's as if you didn't read what I just wrote. Regarding the Soviet records of infiltration, several books have been written on these records and their revelations.

[jjoonathan]

I have zero sympathy for the view that people who disagree with current social/economic/espionage policy should be removed from office on that basis alone. On the contrary, such disagreement is central to how democracy works and those who would threaten it are the true threats to our society.

That said, we continue to disagree about the numbers, and I would like to know the resolution. Could you expand on your claim that "KGB/NKVD archives reveal they had hundreds of agents in the executive branch in the 1940s"? Did McCarthy have a very high false negative rate (thereby finding only 9 of the hundreds of spies)? Are you counting "security risks" as "agents"? What gives?

[aurelian]

Communists, meaning people who were members of the Communist Party or identified as Communists and associated with other Communists, were more likely to have come into contact with Soviet agents and been persuaded to spy or subvert US aims. We are talking about security risks not security certainties. It's entirely reasonable to remove risky people from sensitive positions. Any organization that failed to do so would fail to achieve the objectives given to it through the democratic process; it would serve another master. This is entirely separate from the question of who should choose the objectives of the government.

Ironically, Communists wished to overthrow the US government and were anti-democratic.

[A_COMPUTER]

>I have zero sympathy for the view that people who disagree with current social/economic/espionage policy should be removed from office on that basis alone.

Why? They'll just do a shitty job anyway. People rightfully get up in arms when Republicans want to see an opponent of the EPA appointed to the EPA, or department of education, because they know why it's being done, to undermine it. HUAC was set up to rout out Nazi sympathizers before it started going after communism, should they have been left alone?

[tomjen3]

I don't get why you haven't been downvoted to hell yet: removing people from their positions for the views they hold is _unamerican_.

As an entirely aside if he identified a few real spies and missed hundreds doesn't that suggest he was ineffective, to the point of actual harm?

[aurelian]

Removing enemy agents from positions is not unAmerican.

Removing people with known security risks from sensitive positions is not unAmerican.

He missed hundreds of agents, identified several agents and many more security risks, and spurred heightened security awareness that was a very good thing at the time. Expecting him to know the identity of every Soviet agent before attempting to remove any security risks is unreasonable.

[dalke]

I would qualify that to allow certain bona fide exceptions.

If a nuclear missile launch team has the position that nuclear weapons are unjustified and that pacifistic solutions are the only correct solution to national disagreements, then I think it's okay to remove them from their position, even if they've never acted on their views by not launching when the launch order came.

[SamReidHughes]

> removing people from their positions for the views they hold is _unamerican_.

If you're going to argue about a hiring preference being "unamerican" or not instead of its inherent morality, I don't know what game you're playing. Did you get your posting techniques from the Simple Sabotage Field Manual?

[cafard]

McCarthy was a showman, not a serious investigator. In the long run, he did more harm to the cause he nominally worked for.

[chrisbennet]

Edit: Someone is wrong on the Internet and that person is me. ;-) When I quoted the 3,370 number I missed the "(end-of-fiscal-year count, excluding Postal Service, in thousands)

[ChrisIsWrong] The number of employees in the executive branch in the 1940's peaked at 3,370. [1] So by "hundreds of agents in the executive branch" do you mean that say, 10% of the executive branch were communist agents? [/ChrisIsWrong]

My Dad remembers McCarthy and his famous speech but doesn't know that he was discredited or even what the term "McCartheism" means.

[1] http://www.opm.gov/policy-data-oversight/data-analysis-docum...

[twoodfin]

That number doesn't pass the smell test. Today's Treasury Department alone employs more than 115,000 people[1].

The figures are "in thousands". The Executive Branch at the time had over 3.3 million employees.

[1] http://en.wikipedia.org/wiki/United_States_Department_of_the...

[aurelian]

According to your link, the Department of Justice, which included the FBI, had fewer than 20 employees.

Edit: Thanks for acknowledging the mistake. I made the same one here.

[ska]

In this, as in all classification schemes, there is a real risk of falling into a "dropping FN will raise FP" category error. The core issue may be be that your algorithm is bad, and you can find another with bother better FN and better FP.

[CamperBob2]

Are you referring to the Venona intercepts? Those were from WWII, not the 1950s, weren't they?