Demonstration research and meta-analysis in parapsychology

Meta-analysts use, as their data, statistics that are typically available in relevant published articles, such as means, variances, t tests, and F ratios. As the name implies, they involve an analysis of analyses. The pivotal statistic in meta-analysis, based on the statistics more typically given in published articles, is effect size. One of several ways to determine effect size is to divide the mean difference between a treated group and a control group by the standard deviation of the control group. Effect size, thus, is a standardized mean difference that reflects the magnitude of an effect and hence permits comparison across many studies using divergent procedures and measures (Smith & Glass, 1977).

Meta-analysis is an approach to interpreting psychological data that is increasingly popular yet still controversial. On the one hand, it evaluates the level of support for an effect; on the other hand, it restricts its evaluation to those areas that have been studied in similar ways. Nevertheless, meta-analysis is less subjective than other approaches to combining results across studies; it enables researchers to know if a finding from a study or a group of studies is generalizable to different settings and subjects. The techniques involve gathering data from many studies done on the same topic, statistically analyzing them together, and making a general statement about what the studies indicate. Meta-analysis can also look at how much the relations between factors in one study differ from the relation found in a group of studies that examine the same topic. These statistical procedures allow a number of different experiments of a particular type to be combined if there is any demonstrable effect, and they deal with qualitative issues by correlating effect sizes and quality codings.

Some writers feel that meta-analysis can correct for artifacts (e.g., dichotomization of continuous variables, imperfect construct validity, attrition), assuming that if the meta-analyst is sufficiently diligent in identifying and correcting for artifacts, all the variability in the true effect sizes will disappear (Hunter & Schmidt, 1990). It is important to note that other statistical analyses are available to summarize a series of several studies (e.g., Cooper, 1989; Rosenthal, 1978). For example, an examination of over 80 experimental studies of psi performance in induced internal attention states found combined significance to be 6 x 10 (to the -12th power) for 16 meditation studies, 1.2 x 10 (to the -9th power) for 13 induced relaxation studies, 2.1 x 10 (to the -9th power) for 16 homogeneous visual field (i.e., ganzfeld) studies, and 7.5 X 10 (to the -11th power) for 42 hypnosis studies (Honorton, 1977).

Meta-analyses have been used with increasing frequency in psychology. For example, a meta-analysis of 21 studies of mental practice (e.g., imagery rehearsal for athletic achievement) resulted in an overall average effect size of .68, "indicating that there is a significant benefit to performance of using mental practice over no practice" (Hinshaw, 1991-1992, p. 3). An earlier report by the U.S. National Academy of Sciences had concluded: "The research generally indicates that mental practice accounts for about half a standard deviation in performance gain over what is observed for controls" (Druckman & Swets, 1988, p. 70). This same report found "no scientific justification from research conducted over a period of 130 years for the existence of psi phenomena". However, the authors unaccountably failed to use meta-analyses when evaluating psi phenomena, although they made competent use of meta-analyses in the other topics they evaluated. A reply to this report by several PA members (Palmer, Honorton, & Utts, 1989) used meta-analysis results as one of several ways of countering the Academy's conclusions.