Content provided in partnership with

A blunderbus approach to criticism of statistics. . - book review

Skeptical Inquirer, Sept-Oct, 2002 by Terence Hines

Junk Science Judo. By Steven J. Milloy. CATO Institute, 2001. ISBN 1-930865-12-0. 215 pp. Hardcover, $18.95.

Junk Science Judo is an annoying and shallow book that will not provide the reader with anything like a full account of the problems with junk science and the means to combat it. The author discusses some of the usual outrages of the health hysteria mongers such as claims that cell phones cause brain cancer and that Alar was a carcinogen. But the book is oddly incomplete in this regard. Missing is any coverage of the claims that power lines cause cancer or that breast implants result in immune diseases. These are two of the clearest cases of hysteria-mongering, and their inclusion would have made the book much more compelling by giving crystal-clear examples of why the media-induced fears are unfounded. Oddly, the real target of this book seems to be a poorly defined boogeyman labeled "statistics." Certainly, sloppy use and interpretation of statistics is one of the major problems at the heart of unfounded health fears. A book that would clearly explain the often subtle statistical errors made by promoters of health fears would be a valuable contribution. Unfortunately, this is nor that book.

Most Popular Articles in Reference: The importance of understanding organizational culture; Credit card attitudes and behaviors of college students; What factors attract foreign direct investment?; Libraries Need Relationship Marketing - mutual interest marketing concept, ...; How to set performance goals: employee reviews are more than annual critiques; More »

Milloy's crude broad-brush condemnation of statistics is nicely summed up by his oft-repeated phrase "statistics aren't science." This is like saying "tools aren't carpentry." True enough, but one is going to get damn little carpentry done without tools, even if tools can (duh!) be abused and used incorrectly. The real problem is that Milloy, as judged from his writing, simply doesn't understand statistical techniques well enough to be able to write cogent criticisms of the poor statistical techniques used to support various health scares. In most cases when he uses the term statistics" he really means "correlations." For example, on page 59 he stares, "No study that reports only statistical results can prove a cause-and-effect relationship" (emphasis added). This is a simply absurd statement and would never be made by an author who had even a basic knowledge of statistical procedures. There is an entire class of statistical analyses, called inferential statistics, which are designed precisely to allow resea rchers to draw causal inferences. Milloy's comment would be true if he substituted "correlational" for "statistical." It is certainly true that the finding of a correlation (the term doesn't even appear in the book's index) between two variables does not allow one to conclude that there is a causal relationship between the two correlated variables. But Milloy's ignorance of this fundamental difference between correlations and inferential statistics renders his argument confusing, to say the least.

At other times, Milloy uses the term statistics" in slightly different ways. While he never discusses what inferential statistics are, and how they can be used properly to draw conclusions, he does discuss (in chapter 6) the concept of statistical significance and p values. A p value is a probability value ranging from 0 to 1 that gives the probability of the obtained result being due to chance. The higher the p value, the more likely the obtained result is to have been the result of chance factors. Putting it the other way around, the smaller the p value, the greater the probability is that the results are not due to chance factors--that they are due to the factors manipulated in the experiment. By general agreement, a p value of .05 or less is accepted as a "significant" result. When discussing p values (p. 108) Milloy makes another absurd statement: "How researchers calculate the p-value is not important." This is like saying, going back to the tool and carpentry analogy, that it doesn't matter how a carp enter makes a hole in the wall--a bulldozer is as good as a skill saw. In fact, there are hundreds of different statistical procedures that can be used to calculate a p value. Deciding which one to use on a particular set of data is far from a trivial problem. It is a problem that those of us who teach statistics probably spend more time on in class than anything else. If you do use the wrong procedure, you'll get a p value that is simply wrong. And you will often be badly misled as to what your results really mean. I have found (Hines 1998, 2001) that using the wrong procedure to calculate a p value is very common in experiments that claim to support various pseudosciences.

The author's ignorance of the real nature of, and problems with, statistics time and again prevents him from making his arguments against various health scares as effective as they should be. In chapter 11, "Tricks Are for Kids," he discusses studies that claim to show that exposure to this or that substance causes cancer. A common procedure in such studies is to examine the relationship between exposure and the rate of numerous different types of cancers (or other ailments). When one, or maybe two, significant relationships are found, the "fact" that substance X "causes" cancer type Y is certified. This is the problem of multiple comparisons. If we accept a p value of .05 as indicating significance, we are also accepting that 5 percent of the time, a result will be "significant" simply by chance alone. So, if you look at the effects of exposure to, say, postage stamps on 120 different types of cancer even if postage stamps do not cause cancer, for 5 percent of the cancer types examined there will be a "sign ificant" relationship. Now, of course, on average three of these relationships will show that postage stamp exposure increases the risks of the cancers and three will show that exposure decreases the risks. The decreased risk findings just never get any press. This sort of serious statistical blunder is especially serious in studies claiming to link power lines to various cancers and PCB pollution to developmental disorders (Hines 2002). But Milloy barely mentions this problem.

A blunderbus approach to criticism of statistics. . - book review

Skeptical Inquirer, Sept-Oct, 2002 by Terence Hines

Find Research Guides for:

Find Featured Titles for: Reference

Most Popular Articles in Reference

Most Popular Publications in Reference