SPELLING OUT YOUR LOSS FUNCTION. In statistical inference, there are two serious mistakes a researcher can make. The first type, which practitioners call Type I error, involves the rejection of a true null hypothesis. In many circumstances, people will describe such a case as a "false positive," because the null hypothesis is commonly absence of something (a disease, a price effect) and its rejection involves a mistaken diagnosis or a "statistically significant" coefficient (particularly troublesome if said coefficient also has the "wrong sign.")

The second type, per corollary a Type II error involves the failure to reject a false null hypothesis. Here, a sick person tests as healthy, or a coefficient that ought to differ from zero prints out as "statistically insignificant."

The current political tussle about the September 11 hearings and Saddam's missing weapons involves a confusion over the types of error in inference. Here is Dean at Dean's World claiming to see some careless thinking.
Time to open your eyes and ears. You can't blame people for not stopping something bad (9/11) with bad/vague intel, then complain that they are stopping something bad (Iraq) with bad intel, saying there's "no proof, they didn't do anything to us."
True up to a point. The policy maker, and the voter, must decide what their loss functions are in case of error. To the extent that Saddam's weapons programs were a cassus belli, their absence after the fact has the potential to be a Type I error. (I use the hedge, "potential," to forestall any angry emails alleging those weapons being secreted in Syria or poured into the Euphrates river.) Type I errors are not by themselves grounds for condemning a policy, but policy makers have an obligation to spell out their loss functions. Defenders of the Iraq campaign, for example, might argue that they would prefer to explain why the intelligence was faulty once Saddam is in jail, against the alternative of explaining to their children why they now have to pray with their noses pointed south of east. Their loss function is biased in favor of acting on a positive, whether false or not Critics of the campaign, on the other hand, might argue that future claims of gathering danger are less credible. Their loss function is biased in favor of not acting on a negative, including a false positive.

The September 11 attacks, on the other hand, are an instance of a Type II error. Such error is a bit more difficult to deal with, as much of the statistical inference treats the false positives as the more serious error, and investigators seek to find the most powerful test against the type II error given a small risk of a type I error, or a desire to avoid false positives.

If memory serves, the August 2001 the threat board came up empty, or with indicators of an attack on Japanese interests, when the real threat was within our borders. Critics of the intelligence activities (of several Administrations) are tussling with the loss function they wish to specify. Some would prefer not to explain why there is a gaping hole in Battery Park: their loss function is biased in favor of acting, although that puts them at risk of acting on a false positive. Some would prefer not to act in the absence of a clearly true positive: this increases the risk that there will be a hole in Battery Park. There is, however, evidence that intelligence-gathering activities involving both al Qaeda and Saddam's Iraq missed both positive and negative signals. Part of the art of designing inference that is more powerful against both Type I and Type II errors is understanding the failures of the existing intelligence activities. Two counterfactual columns, by Gregg Easterbrook and by Kathleen Parker illustrate yet another pitfall: suppose that in late August or early September of 2001, the U.S.military had rolled up the September 11 plot and clobbered Afghanistan. As the Twin Towers would still be standing, would that be evidence of a Type I error on the part of the Bush Administration, or would the critics be guilty of committing a Type II error by inferring that nothing was wrong?

(Thanks to Betsy's Page for linking these counterfactuals and other columns in a related vein.)

No comments: