Content provided in partnership with
Technology Industry
Industry: Email Alert RSS FeedPeer review under fire: science's gatekeeping system may need an overhaul
Science News, June 22, 1991 by Bruce Bower
Perhaps nothing arouses scientists' professional passions as much as peer review. After all, this process -- a gatekeeping and quality-control system run by journal editors, independent referees and research program directors -- determines which projects receive funding and which studies get published in what journals. When institutional bouncers rough up your cherished theory and dump your career aspirations on the doorstep, the wounds go deep. And even if the bouncers show respect, they may still inspire fear and doubt with each new grant proposal or manuscript submission.
- Most Popular Articles in Reference
- The importance of understanding organizational culture
- Credit card attitudes and behaviors of college students
- What factors attract foreign direct investment?
- Libraries Need Relationship Marketing - mutual interest marketing concept, ...
- How to set performance goals: employee reviews are more than annual critiques
- More »
Despite its crucial role in this era of "publish or perish," scientific peer review today limps along with its own disabling wounds, asserts Domenic V. Cicchetti, a psychologist with the Veterans Administration Medical Center in West Haven, Conn. In his comparative review of peer-review studies conducted over the past 20 years by various researchers, Cicchetti finds consistently low agreement among referees about the quality of manuscript submissions and grant proposals in psychology, sociology, medicine and physics.
These rampant reviewer rifts represent more than an inevitable clash of perspectives on the worth of any particular study, Cicchetti maintains. Scientists must act quickly to improve the quality of peer review, particularly "the rather arbitrary rejection of grant submissions [that] may prevent or seriously delay the implementation of worthy research endeavors," he argues.
Cicchetti examined studies of peer recommendations and publication decisions made by 18 scientific journals, including the JOURNAL OF ABNORMAL PSYCHOLOGY, AMERICAN SOCIOLOGICAL REVIEW, NEW ENGLAND JOURNAL OF MEDICINE and PHYSICAL REVIEW. Additional data came from grant reviews in chemistry, economics and physics conducted during the 1970s and 1980s at the National Science Foundation and the National Academy of Sciences.
Cicchetti's findings and his recommendations for pumping fresh life into peer review appear in the March BEHAVIORAL AND BRAIN SCIENCES, along with 34 responses solicited from researchers and editorial decision-makers. Many support Cicchetti's proposed reforms, although they differ in their interpretations of the importance of disagreements among reviewers evaluating the same paper or proposal. Others reject the reforms, arguing that referee agreement -- known as "reliability" when analyzed with statistics -- has nothing to do with quality peer review.
Cicchetti remains undaunted. In addition to noting generally poor reliability among referees, he finds them more likely to agree about which grant proposals do not deserve support than about which proposals have scientific value. The same pattern applies to manuscript submissions in disciplines with a general focus, such as general physics, cultural anthropology, social psychology and broad fields of medicine such as cardiology and psychiatry, he says.
Conversely, in disciplines with a specific focus -- including nuclear physics, physical anthropology, experimental psychology and biological specialty areas such as physiological zoology -- Cicchetti finds that reviewers agree more often about worthy manuscript submissions and less often about rejects.
Journals dealing with specific areas of scientific inquiry have higher article acceptance rates and use fewer referees than journals covering general disciplines, Cicchetti points out. Editors of the more specialized publications usually rely on only one initial reviewer; a "thumbs up" review sparks publication approval from the editor, pending suggested revisions, while a "thumbs down" leads to solicitation of one or more additional reviews. General-focus journals often solicit two independent reviews; editors reject a manuscript with two negative reviews, solicit a third evaluation for split reviews and accept papers with two favorable reviews, pending revisions.
Grant proposals receive at least two independent reviews, and sometimes five or more evaluations. Reviewers rate the quality of a proposal on a numerical scale that varies from one funding outlet to another. All the scales contain a strict cutoff score for grants worthy of funding.
Scores often cluster just above or below the cutoff, Cicchetti notes. The inability of referees to agree on the relative quality of these borderline proposals justifies concerns that many worthy projects never win funding, victimized by the draw of reviewers or the proposal's emphasis on a relatively new or unfashionable realm of research, he says.
As for publishing research results, Cicchetti harbors less concern for rebuffed manuscript authors in both general and specific areas of social and medical science, since these researchers invariably succeed in publishing their rejected articles in other journals, often with few or no changes. In contrast, authors in the physical sciences usually do not submit rejected articles elsewhere, apparently regarding the initial peer verdict as decisive, Cicchetti says.
Find Research Guides for:
Find Featured Titles for: News
Most Popular Articles in Reference
- The importance of understanding organizational culture
- Credit card attitudes and behaviors of college students
- What factors attract foreign direct investment?
- Libraries Need Relationship Marketing - mutual interest marketing concept, ...
- How to set performance goals: employee reviews are more than annual critiques
- More »