Lord's Paradox and the Assessment of Change During College

Figure 2, which is based on Lord's original example of weight gain, shows how the hybrid model is a transformation of the analysis of covariance. In Figure 2, the X axis is a student's beginning weight, and the Y axis is a student's ending weight, minus beginning weight. The 45-degree line from Lord's original example now lies along the X axis (Post - Pre = 0, where Post = Pre). The within-group regression lines for men and women now have a negative slope (β 1), but they are still parallel, and the distance between the lines is the same as in the original example.

CONCLUSION

In their tribute to Fredric Lord, Holland and Rubin (1983) noted that the basis for Lord's Paradox is that an analysis of difference scores and an analysis of covariance are designed to answer different questions. An analysis of difference scores answers the question about whether students changed from the pretest to the posttest, whereas an analysis of covariance answers the question of whether students who have the same pretest scores will have different posttest scores. These are not the same questions and it is unrealistic to expect them to provide the same answers.

Personally, I find the approach suggested by Pascarella and his colleagues (i.e., the analysis of covariance) to be a useful framework for assessment research, and I have used this approach in many of my studies of student learning and development. However, the approach described by Pascarella et al. (2003) is not equivalent to an analysis of difference scores, and there is no statistical basis for judging it to be superior to a difference-score study. My preference for the analysis of covariance is based on the theoretically grounded assumption that change during college is the product of many interrelated factors. The weakness of my research is that I sometimes claim to have identified factors that influence how students change, rather than admitting that I have identified how groups differ when precollege characteristics and college experiences are held constant.

REFERENCES

Astin, A. W. (1987). Assessment, value-added, and educational excellence. In D. F. Halpern (Ed.), Student outcomes assessment: What institutions stand to gain? (New Directions in Higher Education No. 59, pp. 89-107). San Francisco: Jossey-Bass.

Astin, A. W., & Ewell, P. T. (1985). The value-added debate ... continued. AAHE Bulletin, 37(8), 11-13.

Baird, L. L. (1988). Value added: Using student gains as yardsticks of learning. In C. Adelman (Ed.), Performance and judgment: Essays on principles and practice in the assessment of college student learning (pp. 205-216). Washington, DC: U. S. Government Printing Office.

Campbell, D. T., & Erlebacher, A. (1970). How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful. In J. Hellmuth (Ed.), Disadvantaged child (Vol. 3, pp. 185-210). New York: Brunner/Mazel.

Ewell, P. T. (2002). An emerging scholarship: A brief history of assessment. In T. W. Banta (Ed.), Building a scholarship of assessment (pp. 3-25). San Francisco: Jossey-Bass.