W

hat do Godzilla and scientific peer review have in common? Each by rights ought to be collapsing under its own weight. Yet somehow they stand.

The good news — for movie fans and scholars alike — is that the behemoths appear to be stable for the moment. But, in the case of peer review, at least, it wouldn’t take much for the system to become perilously imbalanced.

As it stands now, according to a new study, the pool of peer reviewers is able to keep up with the massive number of new papers published each year in biomedicine — more than 1 million, and climbing.

advertisement

The amount of time reviewers spend on the task is, simply put, gargantuan: an estimated 63.4 million hours in 2015 alone, according to the authors of the study, or roughly 7,300 years’ worth of reviewing. If journals were to compensate those reviewers at a reasonable rate of $75 per hour, that’s on the order of $4.5 billion of labor.

But the problem is, the distribution of labor is highly uneven. The researchers found that 5 percent of reviewers were responsible for nearly 30 percent of those hours. And that could lead to shoddier reviews, they wrote: “These ‘peer-review heroes’ may be overworked, with risk of downgraded peer-review standards.”

The paper, published in PLOS ONE, included a few other interesting nuggets about the system. Researchers in the United States, for example, punch above their weight, reviewing a greater share of the world’s papers than they publish. On the other hand, Chinese scientists are net importers of reviewers’ time, publishing more globally than they review — twice as much, in fact. A general rule of thumb that many scientists use is to review one paper for every one they submit.

As we and others have argued, peer review is a deeply flawed system, but one that deserves fixing, not scrapping. The latest study does nothing to change that view. It does, however, point to a few simple changes that could go a long way toward shoring up the structure.

The first is paying reviewers for their time. Although the risk is that the “peer-review heroes” might get a windfall, it’s likely that many scientists who would like to review but feel they can’t spare the time will decide a few extra bucks is worth the extra work.

That could bring about a class of professional reviewers — recently retired academics, perhaps, who bring a wealth of experience to the task, or consultants like Jonas Ranstam, who recently won an award as the world’s most prolific reviewer.

Publishing fewer papers would be another means of rebalancing the load, as Tricia Serio, a biologist at the University of Arizona, argued in a post on the Conversation. “The first step is to reset and clearly state our standards for quality in both publishing and peer reviewing,” she wrote. “The outcome will certainly be fewer publications in biomedicine, but their individual impact will be greater.”

Perhaps. But another solution would be to open the spigot for more and different forms of peer review. Sites such as PubPeer.com and PubMed Commons allow for peer reviewers to participate in ways that are less time-consuming but, in aggregate, far more valuable than the anonymous pre-publication process that happens these days. The old guard has resisted that trend, just as those who may not live until climate change has disastrous effects might ignore those realities, too. But they’re not the ones who will have to turn down the coming tsunami of peer review requests.

Leave a Comment

Please enter your name.
Please enter a comment.

  • I recall having a discussion with my PhD advisor at the University of Queensland, Australia, about peer-review. She maintained that it is essential for ‘good’ science. I responded: “What if the person who presented the new ideas had NO PEERS? (One only has to look at the history of science to realize that every break-through was by a person who had no peers. This leaves peer-review with the role of maintaining the status-quo, which is of course a death sentence for science.)

    • The problem isn’t only the novelty of the solution; good reasoning isn’t novel. It’s that in many and perhaps all fields at least there has been selection for individuals who get results in a PR sense but don’t have the habit, the ability or the time to think things through.

  • Of course people should try and figure out ways to improve the process, but I fear that local fixes (fixing peer review per se) won’t affect the global problem, that conditions are currently not (generally) conducive to scientific imagination, cooperation and integrity. Science is an activity that works best when people are at their free-est and best.

Sign up for our Morning Rounds newsletter

Your daily dose of news in health and medicine