Jonas Ranstam is a peer-review machine.
Ranstam, a medical physicist in Sweden, reviewed 661 papers across 16 scientific fields between Oct. 1, 2015 and Sept. 17 of this year — nearly two per day over that period.
A year ago, such output might have earned Ranstam perhaps a note or two of thanks from the editors whose journals he served. Today, however, he got a bit more — the title of the world’s top peer reviewer.
The inaugural Sentinel of Science awards are the creation of Publons, a UK company that wants to give peer review a bit more luster by showcasing the contributions of reviewers. Publons created the prizes “to honour the expert peer reviewers and editors who stand guard over research quality, and lead the charge for better, faster science.”
That is a positive sign. Peer reviewers are almost entirely uncompensated and usually anonymous, but they play a crucial role in science’s self-correcting process. In recent years, more journals are experimenting with ways to acknowledge these volunteers more prominently.
The financial component of the Publons award is modest. Out of a $2,250 pot Ranstam will receive $250, along with a $1,000 voucher to publish in a journal put out by Thieme, an open-access publisher. On the editor side, the award went to Jose Florencio Lapeña, a pediatric head and neck surgeon at the University of the Philippines who, as a journal editor and member of some editorial boards, handled the most papers during the year. Lapeña will receive $100 cash and a $200 credit with Wiley.
Publons also awards digital “badges” to reviewers who earn a certain level of “merit” points through their reviews; those points can come from publishing the content of their review, or by having their reviews up-voted.
The tallies themselves come from a sort of report-and-verify system. Researchers sign up for Publons, and add review records to their profiles on the site. Publons then works with publishers to verify that they were actually done as claimed. Only verified reviews were counted in the awards, Publons tells us.
Part of Ranstam’s prolific output might owe to his current professional circumstances. Ranstam, who left a position at Lund University several years ago to work as a freelance statistician, said he does his reviewing in the morning, then spends the afternoon on his own projects. “I often work in the evening and usually on Saturdays, but this is a structure that suits me and my wife. My salary is lower, but my quality of life much higher.”
That flexibility means Ranstam can devote more time to what he has long considered a crucial part of the scientific process. At the same time, he worries that attaching financial gain and social praise to the process could dilute the quality of reviews over the long run. Indeed, rewarding quantity may solve short-term needs but be dangerous in the long-term.
“I believe that it is increasingly difficult to find reviewers for the growing number of manuscripts that are submitted, and that awarding reviewers can be a way to raise the interest for reviewing. This would of course be good if it succeeded,” Ranstam told STAT. “On the other hand, it is well-known that rewards change motivation, and this could perhaps be bad. Voluntary contributions have a special value. Will reviews from reviewers pressed to review by competition with colleagues have the same quality and relevance?”
Publons is not alone in trying to find ways of hat-tipping reviewers. The New England Journal of Medicine and the Journal of the American Medical Association, have been issuing shout-outs to reviewers for years. Elsevier will begin recognizing those who get reviews in on time. The journal Collabra is making money part of the recognition by giving small payments to authors and editors either in cash or in credits against future publication fees. And the UK publisher Veruscript says it will pay its reviewers, too.
And Publons’s inaugural awards lineup does have some shortcomings. Although the list of Sentinels is international, it’s plagued by an alarming lack of gender diversity. Publons tells us they didn’t collect gender data, but by our count, only 1 of the 44 named winners, Ana-Maria Florea, a cell biologist in Germany, is a woman. Whether that’s because men are overrepresented at Publons, among peer reviewers, or both, we’re not sure, but it sure would be good to recognize more women for the reviews they do.
For Ranstam — and, presumably, other high-volume reviewers — although recognition for the efforts is gratifying, the process has its own rewards. Ranstam says reading all those articles is a good way for him to stay up on the literature and keep his own skills sharp. “And it may seem strange, but to understand the underlying causes or reasons for a methodological misunderstanding, and trying to explain this to the author, is personally developing. I would have been a much better lecturer 20 years ago, if I then had had the reviewing experience I have now.”
This is nearly the same questions raised before by Dr GS Mukherjee: Are there any journals that pay reviewers? Why don’t researchers request payment for refereeing? Why don’t researchers get more academic credit for refereeing? Was there any study on whether money alters the content or biases of the reviews?
It is true that most reviewers (actually, nearly all) do not get paid. Normally, editors respond, “Sorry, we have no budget to pay for refereeing”. Peer reviewers are asked to volunteer, and they volunteer because they know that someone else would have to review their papers too. One of the most criticized aspects of the current publishing scheme is that academics do pretty much all the work for free and publishers get the money. If you don’t get paid for the papers you write, it’s weird to get paid for the papers you referee. Writing a paper is certainly worth some academic currency, it just doesn’t come out directly in dollars. However, credit given for papers refereed is moderate to nebulous to non-existent. That’s what could be fixed. It is said to be an altruistic act that would be considered to be prestigious. But, there is a journal that functions differently. It is called Collabra, you may read the news about it. In addition, in some fields, for some journals, they are paid a fee. This approach seems to be most popular with economics journals – as they are heavily into studying the way people can be motivated by money, I suppose this makes sense! An exceptional example is the Journal of Financial Economics – most articles have a single reviewer, who is paid $275 cash (and a ~$200 discount off their next submission), contingent on a timely review. Lower down the scale, the Journal of Banking and Finance talks about offering “tokens of appreciation” for reviewers; there’s no cash value given but the phrasing suggests it’s probably a good bit lower. Now, why do these journals do it? Probably because they always have done, an explanation which applies to a lot of strange quirks of the academic system. But is it a good idea? The recent proposal by Scientific Reports to have a two-track paid- and unpaid-peer-review system was incredibly contentious, after all…
The Journal of Public Economics recently tested the system out – they took 1500 review requests for their papers, and divided them into four groups:
1. a six-week deadline, but no penalty for missing it
2. a four-week deadline, but no penalty for missing it
3. a four-week deadline, and a promise of $100 payment for meeting the deadline
4. a six-week deadline, but reviewers told turnaround times would be made public
All variants worked well. The group with a four-week deadline had an average turnaround time of twelve days less than the six-week deadline group. Payments took another eight days off the turnaround time, and “public credit” another 2.5 days. The money/credit groups wrote slightly shorter reports, presumably as they were more motivated to make a hard deadline, but the editors did not see them as of noticeably lower quality. So… yes, payment can work. But it relies on the journal having the money (two reviewers is $200/paper), and – intriguingly – it’s not quite as effective as the no-money-needed option of just giving a shorter deadline. And even when it does work, it only makes sense if done systematically by the journal. Individuals asking to negotiate their own review payments is unlikely to work in the same way.
By way, there seems to be a company, Rubriq, which offers paid peer-review services. I don’t know who their current clients are, if any. There are others who do this feature too. There are several concerns on why this should be paid as reviews do devote their precious time for this. The fact that they continue to do so for free is what still makes the publishers feel it is fine not to pay for them. As long as altruistic reviewers (well, in the case of money) exist, journals will continue to function without paying the reviewers. A couple of years ago Scientific Reports initiated a project where authors could pay for fast-track peer review, guaranteed within two weeks through Rubriq. The editorial board revolted over the decision and threatened to quit. They argued that doing this would create a peer-review market driven by profit, mostly with undesirable effects on the quality of reviews (reviewers are not selected based on their expertise), and that it would create a two-tier system where researchers who can afford this service would have an unfair advantage in publishing. Finally, the project was canceled.
Academics do most of the work – Most academics wouldn’t object to this state of the affairs. Of course, academics do most of the work in publishing (especially reviewing) – they’re the ones who are qualified to do it. You can’t have some bureaucrat take care of reviewing the work; you need someone who knows the field.
Publishers do a comparatively small amount of work – Academics might grumble at this, but there’s a comparatively little that the publishers are qualified to do. Typesetting, printing, maintaining the journal website, administration in the reviewing process … and that’s about it. All the actual content decisions have to be done by knowledgeable people (academics). There’s certainly some journals which try to offload things like typesetting onto the authors, but in part that’s financially driven …
Publishers get the money – This is the main point of upset. Publishers charge what is viewed as an excessive amount. … but it’s not that they’re charging money per se, it’s more that academics lose access to the content due to expense. Most academics were completely satisfied when they had access to (paid) library subscriptions. It’s only when budget cuts (and publisher price increases) caused libraries to cut subscriptions that academics got upset. But again, it’s less having to pay for things and more not being able to access everything they need.
Academics don’t get any money – This is the point you may consider. However, I’d say most academics don’t have a problem with it. Refereeing for a journal is considered by most to be community service – it’s something that needs to happen, and they’re the only ones qualified to do it. It’s quid-pro-quo: others review your articles, and you review other people’s. Attempting to make it a paid-for enterprise makes the person asking for the money seem greedy.
So where does that leave you? Academics (mostly) don’t have a problem doing most of the work – and doing it for free. The complaints are on the publisher’s side: they’re charging too much for what little they do. Demanding that publishers pay academics for reviewing isn’t going to change that. If anything, it will make journal access more expensive, as they now need to pay for reviewers. So charging a fee when reviewing isn’t going to fix the problem, but asking for one is only going to make you look naive and greedy to the (academic) editor who you’re in contact with. Note that there’s a very big difference between “Decline to review based on principles” and “Decline to review … unless you pay me”.
At last, it’s part of a scientific culture that peer-reviewing is done without charge, just as it’s part of a business culture that you don’t wear sagging shorts and torn t-shirts to work.
Is it something that might change? Sure; it’s not a formal contract, it’s a cultural thing, and cultures change. But most scientists feel that peer review is a good thing, and most scientists feel that it should be free, so it’s unlikely that it will change soon.
Keep in mind that, while there’s a lively online debate about the merits of peer review and open-access publication and so on, this is a debate among tiny and non-representative populations of scientists. Don’t confuse the passion and attitudes that you see online with how most scientists actually feel.
What is the peer-review literature?
Wikipedia took the idea of peer-review and applied it to volunteers on a global scale, becoming the most important English reference work in less than 10 years. Yet the cumulative time devoted to creating Wikipedia, something like 100 million hours of human thought, is expended by Americans every weekend, just watching ads. Science advances by trial and error. When mistakes are made, the peer-review publication process usually roots them out. Cuccinelli’s version of the scientific process would be “make an error and go to trial.” Einstein did not arrive at E=mc2 in his first attempt. If he were working in the state of Virginia under Cuccinelli today, he could be jailed for his initial mistakes and perhaps never achieve that landmark equation. So I see, peer-review is a serious exam which can re-evaluate your abilities each time you have been offered to review an article. However, you can say “thanks, I can’t” if the topic doesn’t belong to the field of your expertise.
Before Publons, reviewers were doing their tasks of reviews as Palaeolithic as the ones trying to discover the peer-review dredge. However, the files of the reviewers are collated by Publons which initiates the spirit of rival between researchers.
The main challenges for a reviewer in peer reviewing:
– Knowing the field to which a certain manuscript belongs very well.
– Having experience in reviewing manuscripts.
– Having abilities to make reviewer’s remarks clear.
– Having enough time to evaluate the manuscript in depth.
– Obeying the editorial deadline for doing a review.
– Having a strong interest in scholarly journals.
– Being fluent in English.”
However, the worst problem of peer reviewing is its time-consuming nature.
…. “But recognizing peer review is a worthy cause: The labor it takes to peer review science papers would cost a couple billion dollars a year, if the reviewers were paid for their time, which they almost never are.” – This quote in SLATE speaks volumes of the role of reviewer
2 reviews per day! 16 different fields! Although very impressive, I am very concerned about the quality of these reviews and so should be the editors.
I also found it weird that the article didn’t bring this up. It might be understandable if Ranstam only focuses on the statistcal aspect of the studies, going through the numbers might be his morning sudoku.
Great point about diversity. Our analysis suggests between 20-25% of reviewers in the top 10% for each field are female (this isn’t perfect as we don’t ask researchers to disclose their gender when they sign up). We’d love to see more women getting recognition for their expert reviewing efforts across the world’s journals. Hopefully more begin to track their review contributions on Publons – the largest cross-publisher reviewer platform – so we can give everyone involved the recognition they deserve.