The perils of post-hockery - interpretations of alleged phenomena after the fact

When we engage in post-hoc reasoning, we blur the distinction between the contexts of discovery and verification and risk misplacing our faith in unwarranted beliefs. Some simple steps can avoid the seductive traps of post-hockery.

How do psychologists come to believe in the diagnostic utility of responses to a series of abstract inkblots? How can each student in an entire class believe that a generic character sketch is a custom-tailored description of his or her personality? Individuals may cling tenaciously to these beliefs, despite the scientific and logical evidence that refutes them. Unfortunately, it is relatively simple for anyone with an active imagination, access to a steady supply of ambiguous data, and a willing suspension of critical thinking to "discover" and "validate" personally meaningful relationships that do not actually exist. The key is to focus on post-hoc ("after the fact") interpretations of alleged phenomena that selectively admit evidence.

Strengths and Shortcomings of Human Judgment

"Post-hockery" takes advantage of several interrelated strengths and shortcomings of human judgment. We often blur the crucial distinction between the generation and testing of hypotheses. In the context of discovery, our outstanding pattern-recognition and reasoning abilities are indispensible. We can detect potentially relevant environmental cues and formulate sophisticated hypotheses about underlying causal relationships. However, unaided judgment can fail us in the context of verification. We do not routinely subject our cherished beliefs to rigorous tests, and we often accept the first preferred explanation as fact. Particularly if this explanation is interesting or entertaining, confidence may become unshakable.

One of the surest signs of post-hockery is the inability of its practitioners to predict future events. Although successful prediction does not necessarily imply a deep understanding, the reverse is true. One who truly understands a relationship can use this knowledge to make valid predictions, whereas the post-hoc reasoner can only "explain" past events. Three examples serve to illustrate the importance of prediction in testing claims to knowledge.

The Bible Code

The recent controversy over The Bible Code (Drosnin 1997) stems from post-hoc reasoning. The alleged discovery of God's hidden messages actually takes advantage of post-hockery in two problematic ways. As David Thomas (1997, 1998) has exquisitely detailed, the reported probabilities of finding the messages are computed in a wildly inappropriate manner. If you look in a fantastic number of places, and count anything that you stumble upon as supportive evidence, you are guaranteed to discover meaning where none exists. That the hidden messages of the Bible Code can be found with statistically predictable regularity in a wide array of texts speaks strongly against their profundity.

A more subtle problem with the code is of special relevance here. Important messages are supposedly revealed through the juxtaposition of words such as "Hitler" and "Nazi," "Roswell" and "UFO." Here is a challenge to those who believe that the Bible contains hidden messages: predict something. Promoters of the Bible Code tell us that it can reveal the future, so by all means reveal it to us! If coincidental text alignments are the culprit, however, then "meaning" will be found only after events have transpired, and the predictive failure of the Bible Code will reveal a message that is certainly not hidden.

The "Hot Hand" in Basketball

A second example of post-hockery is the widespread belief in the streak shooting of basketball players. You would be hard-pressed to watch an entire televised game without at least once being told that so-and-so has the "hot hand." There are two distinct ways to understand this remark. It might simply indicate that a player has made a series of shots. Nobody would take exception with this descriptive use of the term. Fans, coaches, and players, however, agree that this is not the intended meaning. The term "hot hand" is used and understood in a predictive sense: the player with the "hot hand" is engaged in streak shooting that is expected to continue.

Unfortunately for its adherents, the "hot hand" label holds no predictive value. Research shows that the "hot hand" marks past success but predicts nothing (Gilovich, Vallone, and Tversky 1985). This unwarranted belief is likely due to false pattern recognition. There does seem to be something magical about a player who has made several successive shots. Fans, coaches, and players alike feel compelled to create a causal explanation rather than to accept the mundane statistical fact that when outcomes are distributed probabilistically, streaks occur by chance far more often than we imagine.(1)

Atheoretical Statistical Analyses

A third example of post-hockery stems from statistical analysis methodologies used by many researchers. Computerized data analysis, which has greatly accelerated the pace of scientific progress, provides massive input to post-hoc analysis. The enormous difficulty involved in hand calculations once restricted complex statistical analysis to explicit tests of hypotheses. These days, a few keystrokes can generate as much statistical output as an army of graduate students operating their pocket calculators all summer long. Correlating just a couple dozen variables with one another will produce a matrix containing nearly 300 correlation coefficients. By convention, results that occur at a level expected by chance just 5 percent of the time are called "statistically significant." We can therefore expect about fifteen spuriously significant correlations within every matrix of 300. Each spurious correlation is grist for the master of post-hockery.