Insights into the scientific gatekeepers: A fight for the status quo?
A new study has found that well respected peer reviewed journals have rejected manuscripts that could discuss outstanding or breakthrough work. The researchers found that some manuscripts rejected by three leading medical journals went on to receive a large number of citations after publication in other journals. The study, which if course was peer reviewed itself, offered insight into the process that the typical researcher might not see.
Most of us understand that the peer review system serves as a gatekeeping system for scientific research. It is designed to ensure the publication of only the most well researched studies with the most important findings, it isn’t perfect, but for the most part it works. So much so, scientists depend on publication of their research in peer reviewed journals for career advancement. While peer review can prevent the publication of unimportant, poorly researched manuscripts, or outrageous unsubstantiated claims, some scholars are concerned that it actually protects the status quo and suppresses innovation.
To evaluate this claim, the team studied a dataset of manuscripts submitted to Annals of Internal Medicine, British Medical Journal and The Lancet in 2003 and 2004. These journals rejected 946 of the 1,008 manuscripts in the dataset. 722 of the rejected journals never made it past the editor’s desk and therefore, never actually reached the peer review stage (an important distinction), at one or more of these three publications.
Other journals subsequently published 757 of those rejected manuscripts. Then the researchers looked at the number of citations these manuscripts went on to receive. They used the number of citations as a measure of quality, reasoning that when performing their own research, scientists usually choose to build on work they consider of good quality.
The team found that, for the most part, editors and peer reviewers at the three elite journals did a good job of predicting the popularity of particular research papers among scientists. When the researchers assigned numerical scores to evaluations by peer reviewers, they found that, among both accepted and rejected papers, those with lower scores tended to receive fewer citations. Rejected manuscripts tended to receive fewer citations than accepted ones, and desk rejected manuscripts tended to receive fewer citations than those not rejected until the peer review stage.
However, the team discovered that some of the desk rejected manuscripts went on to receive many citations. The elite journals had rejected 14 of the most highly cited manuscripts and had desk rejected 12 of those.
It’s important to mention that the researchers acknowledge that the three journals may have rejected some of the manuscripts because they were more suited to specialist journals, not because of the scientific reasoning (or lack thereof) behind them. Nevertheless, previous research suggests that peer review can incorporate bias, with reviewers basing decisions on the social characteristics of the authors or the intellectual content of the work.
Gatekeepers tend to prefer work closer to their own and to favor the scientific status quo. It is a bias we all share, one that is hard to eliminate when we are using a peer review system. As I said before, it may not be perfect, but it’s the best we have. Understanding the bias in that system can help correct for it and lead to better functioning system.
Siler K, Lee K, & Bero L (2014). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences of the United States of America PMID: 25535380