Content provided in partnership with

Lord's Paradox and the Assessment of Change During College

Journal of College Student Development, May/Jun 2004 by Pike, Gary R

During the 1980s, discussions of "value added" occupied a prominent place in the assessment literature (see Astin, 1987; Astin & Ewell, 1985; Baird, 1988; McMillan, 1988; Pascarella, 1989; Terenzini, 1989; Warren, 1984). Despite widespread agreement about the importance of assessing how students change as a result of college, discussions of value added are rare today, due in large part to the many problems associated with assessing change (Ewell, 2002).

Most Popular Articles in Reference: The importance of understanding organizational culture; Credit card attitudes and behaviors of college students; What factors attract foreign direct investment?; Libraries Need Relationship Marketing - mutual interest marketing concept, ...; How to set performance goals: employee reviews are more than annual critiques; More »

One of the most vexing problems confronting the assessment of value added is that the two methods generally used to analyze change-difference scores and the analysis of covariance-produce contradictory results. In a series of three articles, Frederic Lord (1967, 1969, 1973) compared the results of analyses of pretest-posttest designs that used difference scores and the analysis of covariance. Best known is his example of weight gains in males and females from the beginning to the end of their first year in college. (According to Lord [1973], he chose weight gain to show that the phenomenon he described was not due to measurement error.) The paradox is that analyses using difference scores generally show no statistically significant changes, whereas analyses of covariance do show statistically significant changes.

Lord's Paradox is shown in Figure 1. In the figure, the ellipses represent scatter plots of the beginning and ending weights of men and women during their first year in college. Individuals falling on the 45-degree line in the figure are students whose weights did not change during the school year (i.e., gain = O). The fact that the centroids of the scatter plots for men and women fall on the 45-degree line indicates that there was not a statistically significant difference in mean weight gain for men and women.

The two dashed lines in Figure 1 represent the slopes for regressions of ending weights on beginning weights (i.e., an analysis of covariance). The fact that the two lines are parallel demonstrates that the assumption of homogeneity of the regression functions has been satisfied. Most important, the fact that the intercepts for the two regression lines are substantially different indicates that there are statistically significant differences in weight gain for the two groups. Specifically, males weigh significantly more than women, after controlling for initial weight. Stated differently, for males and females with the same initial weight, males will have significantly higher ending weights.

In his subsequent articles, Lord provided additional examples of the paradox, including examples of how differences in analytic approaches have affected the results of actual program evaluations (see Campbell & Erlebacher, 1970). What remains uncontested is Lord's (1965, p. 305) conclusion from his first article: "with the data usually available for such studies, there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled pre-existing differences between groups."

AN ALTERNATIVE APPROACH?

In the Journal of College Student Development, Pascarella, Wolniak, & Pierson (2003) described an approach they believe provides an alternative to traditional pretest-posttest measures of change. Pascarella and his colleagues advocated regressing the posttest scores on pretest scores and other predictor or control variables. In order to demonstrate the validity of their approach, Pascarella and his colleagues presented an empirical example based on 2,267 students attending 18 four-year colleges and universities that participated in the National Study of Student Learning. The outcome variable in the first model was end-of-first-year degree aspirations and the independent variables were degree aspirations at the beginning of college, gender, ethnicity, socioeconomic status, pre-college academic ability, the average pre-college degree aspirations of all students at the institution, first-year grades, course-related interaction with peers, academic effort and involvement, work responsibilities, interactions with peers not related to courses, and Greek affiliation. In the second analysis, the outcome variable was the difference between end-of-first-year and beginning degree aspirations. The independent variables were the same as in the first model.

Pascarella and his colleagues found that the effect coefficients (bs) in the two models were virtually identical. The only meaningful difference was that prc-collcgc aspirations had a significant positive relationship to endof-first-year degree aspirations in the first model and a statistically significant negative relationship to the difference scores in the second model. This result is to be expected given the tendency for gain scores to be negatively correlated with initial status (Thorndike, 1966). Based on these findings, Pascarella et al. (2003) concluded: "As long as a pretest measure is included as a statistical control, the estimated impacts of all other variables in a regression equation on posttest scores will be essentially the same as their estimated impact on pretest-toposttest gains" (p. 125).

Lord's Paradox and the Assessment of Change During College

Journal of College Student Development, May/Jun 2004 by Pike, Gary R

Find Research Guides for:

Find Featured Titles for: Reference

Most Popular Articles in Reference

Most Popular Publications in Reference