Lord's Paradox and the Assessment of Change During College

Several observations can be made about this research. First, the model in which endof-first-year degree aspirations were regressed on precollege degree aspirations and other variables is an analysis of eovarianee. second, the results of this study seem to contradict Lord's Paradox. The analysis of difference scores (i.e., Model 2) produced results that were virtually identical to an analysis of eovarianee (i.e., Model 1). Third, and possibly explaining the contradiction of Lord's Paradox, the second model differs from a traditional analysis of gain scores in that precollege degree aspirations were included in the second model as a predictor of change.

LORD'S PARADOX REVISITED

To determine whether the inclusion of precollege degree aspirations in the second model accounted for the lack of support for Lord's Paradox, I reanalyzed the data using the correlation matrix and vectors of means and standard deviations provided by Pascarella. In the reanalysis, I included three models. The first two models were identical to those used in the original study. The third model was a traditional difference-score model in which pre-college degree aspirations were not included as an independent variable. The results of the reanalysis are presented in Table 1.

An examination of the coefficients in Table 1 reveals some very minor differences in the results for the first two models and in the results reported by Pascarella, Wolniak, and Pierson. These differences are probably due to use of a correlation matrix in the reanalysis. Despite the differences, it is clear that the results for the first two models are substantively the same and are consistent with the results reported by Pascarella and his colleagues. The results for the third model, a traditional analysis of difference scores, are substantially different. Instead of explaining more than 20% of the variance, the third model explained slightly more than 1% of the variance in the outcome measure. Most statistically significant relationships disappeared in the third model, and the signs (i.e., direction) of some relationships changed as well. Thus, the consistency of results reported by Pascarella et al. (2003) is not consistency in the results for an analysis of covariance and a traditional analysis of difference scores. It is a consistency of results between an analysis of covariance and a hybrid approach involving an analysis of covariance and gain scores. Lord's Paradox is alive and well. Most important, from the available data, it is not possible to determine statistically which approach is correct.

Simply saying that the second model analyzed by Pascarella and his colleagues was not a traditional difference-score model is not a sufficient reason for discounting it. At first glance, the hybrid model appears to offer a double correction for precollege degree aspirations. A closer examination, however, reveals that the hybrid approach is an analysis of covariance with a rescaled outcome variable. Subtracting the value of the pretest from the posttest changes the scale for the outcome, but it has little effect on the within-group regressions that form the basis of the analysis of covariance. The values of the intercepts (a) are changed, but the difference between the intercepts for a given independent variable (e.g., gender) remains the same. The slope (b) of the within-group regression become β - 1, where β is the slope from Model 1.