Should Ganzfeld Research Continue To Be Crucial In The Search For A Replicable Psi Effect? Part I. Discussion Paper And Introductionto An Electronic-Mail Discussion

In early 1997, Richard Wiseman and I, in an attempt to determine whether other experimenters had indeed succeeded in replicating these results under well-controlled conditions, meta-analyzed the 30 published ganzfeld studies conducted since the publication of Hyman and Honorton's methodological guidelines (Milton & Wiseman, 1999a). The studies' combined outcome was not statistically significant, and the mean effect size was near zero (see Table 1). The mean effect size in the recent studies is less than a seventeenth of that found in the PRL work, and a post hoc comparison shows that it is statistically significantly lower than the mean effect sizes of the PRL and earlier ganzfeld databases (see Table A2).

Updating our meta-analysis to include the studies (see Table Al) published to date (March 1999) since our meta-analysis was completed in February 1997 renders the overall cumulation statistically significant, [1] but fails to raise the mean effect size to even a sixth of that obtained in the PRL or earlier ganzfeld studies meta-analyzed by Hyman (1985) and Honorton (1985) (see Table 1). Moreover, the statistical significance of the updated cumulation is due not to renewed success by a range of investigators, but solely to the inclusion of an extremely successful study by Dalton (1997a) (see Table 1). Whether Dalton's study is included or not, it is clear that the effect size obtained in Honorton's autoganzfeld studies and in the earlier ganzfeld database has not replicated. Post hoc comparisons show that the updated database of recent studies, with or without the Dalton study, has a mean effect size statistically signficantly lower than those of the earlier meta-analyses (see Table A2).

The same is true if a variety of alternative outcome calculation and cumulation methods are used to analyse the recent studies rather than the ones that we preplanned and applied (Milton & Wiseman, 1997a). Since the presentation of our meta-analysis at the 1997 Parapsychological Association Annual Convention, a number of colleagues have informally pointed out that using several different methods of calculating or cumulating individual study outcomes, or introducing various criteria for excluding outliers, results in overall statistical significance of varying degrees for the database. Regardless of arguments over the post hoc and possibly selective nature of these analyses, none of them has the effect of raising the mean effect size in the new database by any meaningful amount, because of the relative insensitivity of means compared to the statistical significance of cumulations when slight changes are made in the treatment of a database. For example, using Bem and Honorton's (1994) method to sum the number of direct hits obtained across studies (approximating the number of direct hits from the standard normal deviate of the study's reported outcome measure if direct hits were not reported) results in a total of 331 hits in the 1198 trials in the database. This is a statistically significant outcome, p = .19, one-tailed; but the effect size measured in this way is only .60.