Contribute Try STAT+ Today

Some observers have hailed the response to Covid-19 as a triumph for science. A hesitant triumph, perhaps. Full of missteps and recriminations, frustrations and death. Maybe more a hard-won victory.

Scientific outcomes certainly support some optimism. The world has gained a better understanding of the relevant Covid-19 disease processes, assembled a solid clinical research base for managing the disease, and produced multiple vaccines that are driving down infections and deaths.

Yet the infrastructure for producing empirical knowledge about the SARS-CoV-2 virus frequently failed, leading to potentially devastating public health consequences. Science shot itself in the foot. Not once or twice, but regularly, and painfully. It’s amazing any metatarsals remain. A big part of the problem has been the ever-widening mismatch between the speed of scientific publishing and error correction.

advertisement

The slow pace of correcting errors

Research papers published in scientific journals are both the primary source of public scientific information and the main metric for scientists’ career success. Getting a paper published can be a long, tedious process that involves peer review: a detailed assessment of the manuscript by a small number of external experts.

These experts aren’t infallible, so some published papers contain errors. These may be obvious and egregious, but usually have more consequences for other scientists than the general public — when a study about how to get children to eat more carrots is retracted, it’s mostly an issue for other researchers who have built their work off the findings. Yet even when conclusions could immediately affect public health, scientific publishing maintains an astonishingly strong resistance to modifying or correcting previously published articles. And doing nothing is the most common outcome.

advertisement

If evidence of errors does emerge, the process for correcting or withdrawing a paper tends to be alarmingly long. Late last year, for example, David Cox, the IBM director of the MIT-IBM Watson AI Lab, discovered that his name was included as an author on two papers he had never written. After he wrote to the journals involved, it took almost three months for them to remove his name and the papers themselves. In cases of large-scale research fraud, correction times can be measured in years.

Imagine now that the issue with a manuscript is not a simple matter of retracting a fraudulent paper, but a more complex methodological or statistical problem that undercuts the study’s conclusions. In this context, requests for clarification — or retraction — can languish for years. The process can outlast both the tenure of the responsible editor, resetting the clock on the entire ordeal, or the journal itself can cease publication, leaving an erroneous article in the public domain without oversight, forever.

And that was before Covid-19.

First Opinion Podcast: STAT’s weekly podcast covers the people, issues, and ideas shaping the life sciences writ large. Subscribe today.

The pandemic sped up time to publication

The pandemic turbocharged the slow process of scientific publishing. A tremendous hunger for information arose, along with boundless anxiety, fear and distrust, information and disinformation, overconfidence and caution. The world was blindsided. Answers were required. Early on, it seemed like any answers would do.

Preprint servers — online portals for making public complete but non-peer-reviewed manuscripts — have been part of scientific publishing since the 1990s. In the information-hungry environment of 2020, their use greatly expanded. Likewise, many journals organized formal peer review processes to be completed in days, rather than the weeks or months typical before the pandemic, and then published studies online. Today, a report written by a single researcher or team can go viral minutes after it is released and can change health care policy the next day.

This renaissance in publishing, however, has not shifted the slow process of reconsidering or investigating published papers. The mismatch between the new speed of publication and the old glacial process for correcting errors has become extreme. This mismatch has had very real — and very dangerous — consequences.

The debacle around hydroxychloroquine is the most notorious example of this. The original report was published on a preprint server in mid-March 2020 and then appeared on the journal’s website after a single day’s peer review. Within 72 hours, researchers had raised extremely serious concerns with the study’s ethical approval process, methods, and conclusions.

The original paper was the primary force behind changes in global health policy, with millions of people around the globe being given hydroxychloroquine as a treatment for Covid-19 almost immediately.

Elsevier, the company that published the report, announced an investigation on April 11, 2020. It took three months to complete — an eon in pandemic time — and an independent review concluded that “this study suffers from major methodological shortcomings which make it nearly if not completely uninformative.”

But here’s the problem: The paper touting hydroxychloroquine has more than 4,000 citations according to Google Scholar, while the review concluding it was worthless has been cited just 38 times. Presumably, the state of Oklahoma didn’t read the review or it wouldn’t have stockpiled so much of the drug that it now can’t get rid of.

Another example: A study on pandemic-related school closures concluded that they caused more deaths than Covid-19 itself — a finding sure to grab headlines, but from an analysis tainted by a mathematical error. One of us (G.M.K.), along with a colleague, tried to get it corrected almost immediately after publication, given the impossibility of its conclusion. Yet even though we exhausted every option at our disposal for getting the error corrected, the interminable pace of academic critique meant that two months ticked by before the study was modified. The surviving paper now asserts that school closures for European children were entirely harmless, while in the U.S. they cost more years of life than most other forms of human disease combined.

In the interim, this study was featured on dozens of newscasts (many complete with long-form interviews of the authors), and was cited in official documents produced by both the World Health Organization and the European Union. The subsequent correction, as far as we can tell, had no major media coverage whatsoever.

Another example is a mathematical modeling paper that concluded that stay-at-home policies did not reduce deaths from Covid-19. The controversial finding was immensely popular, but after noting several issues with the research, one of us (G.M.K.) and several colleagues submitted letters to the journal pointing out that the conclusions may not mean very much. After two months, these critics are now revising their comments — which the peer reviewers and editors mostly agree with — for publication as letters.

Meanwhile, the horse has long since bolted from the barn. In the two months since publication, the status of stay-at-home orders has lost a great deal of relevance, as vaccine deployments reduce their necessity. It was tremendously important when published, but the academic process to correct it will last well beyond its immediate relevance.

And this is a best-case scenario. The journal editor was incredibly responsive, putting up a notice of concern on the paper almost immediately and getting back to critics within days. The report’s authors are engaged in the discussion. If this was happening at almost any other time, it would be a delightful example of academic parlay, where disagreements about evidence were hashed out over a metaphorical beer.

But it’s a pandemic. If the criticism of this report is right and the authors wrong, it’s not just an interesting side note in the annals of science but a life-and-death scenario.

And these are just some high-profile examples. One implausible preprint on using vitamin D to treat Covid-19 simply disappeared from the internet when researchers started questioning whether its authors existed at all. Other examples include everything from wild, unsupported hypotheses to potentially fraudulent studies to mathematical models that were simply wrong. The website Retraction Watch has been chronicling the deaths of these withdrawn pieces of research.

The scientific publishing and correction mismatch

This mismatch — fast publication, slow correction — has killed people. Evidence that scientists knew almost immediately to be inaccurate, unsupported, or fraudulent was frequently used to influence or change public health policy.

A phrase often used in introductory courses on the scientific method, “science is self-correcting,” is a barefaced lie. Instead, scientific criticism is devalued, ignored, or openly attacked.

The status of the correction process, the slow, grudging footnotes that provide crucial corrections to the world’s knowledge base, is built in large part from scientists who go out of their way to trigger or provoke reevaluation of published work.

The list of epithets employed for people who attempt it is long and vicious: shameless little bullies, vigilantes, the self-appointed data police, scientific McCarthyites, destructo-critics, and most famously, methodological terrorists. Given this attitude, it should come as no surprise that almost no one around the world is employed to check the quality of scientific work after it is public, as funding to do so is effectively zero.

This form of review is thankless, pedantic work which brings scant rewards; is regarded as academically and scientifically second class; and is universally treated as the purview of antisocial oddballs. Publish terrible research and it can be regarded as solid progress along one’s career path, or even rewarded. Critique such failures and you’ll be ignored at best, and mocked or even threatened, legally and otherwise, at worst.

One of us (G.M.K.) was personally attacked in an academic journal article that included outright falsehoods about him because of his Twitter critiques of published academic papers. The other (J.H.) has been threatened twice with legal action for providing authors with mathematical critiques of their work. These were not allegations of fraud or fabrication. A simplified version would proceed along the lines of “Two plus two does not equal five. Care to explain?” which might draw the response, “You’ll be hearing from my lawyers.”

This situation must change, and change quickly. Any crisis that requires scientific information in a hurry will produce hurried science, and hurried science often includes miscalculated analyses, poor experimental design, inappropriate statistical models, impossible numbers, or even fraud. Having the agility to produce and publicize work like this without having the ability to correct it just as quickly is a curiously persistent oversight in the global scientific enterprise. If corrections occur only long after the research has already been used to treat people across the world, what use are they at all?

There are some small steps in the right direction. The open-source website PubPeer aggregates formal scientific criticism, and when shoddy research makes it into the literature, hordes of critics may leave comments and questions on the site within hours. Twitter, likewise, is often abuzz with spectacular scientific critiques almost as soon as studies go up online.

But these volunteer efforts are not enough. Even when errors are glaring and obvious, the median response from academic journals is to deal with them grudgingly or not at all. Academia in general takes a faintly disapproving tone of crowd-sourced error correction, ignoring the fact that it is often the only mechanism that exists to do this vital work.

Scientific publishing needs to stop treating error-checking as a slightly inconvenient side note and make it a core part of academic research. In a perfect world, entire departmental sections would be dedicated to making sure that published research is correct and reliable. But even a few positions would be a fine start. Young researchers could be given kudos not just for every citation in their Google scholar profile but also for every post-publication review they undertake.

In the lumbering Gargantua that is academic research, it is past time for this hostile, piecemeal system of scientific publishing to be replaced by a formal approach to catching mistakes.

Maybe then we’ll be prepared for the next crisis.

Gideon Meyerowitz-Katz is an epidemiologist from the University of Wollongong in New South Wales, Australia. James Heathers is the chief scientific officer of Cipher Skin.

  • While this statement is obviously hyperbolic, “Science shot itself in the foot. Not once or twice, but regularly, and painfully. It’s amazing any metatarsals remain.” I still dig it. Congratulations if it is original with one of you guys–I’m going to steal it sometime in the future.

  • The science game, being a human enterprise, is never (and will never be) free of jockeying for status. The “self” in science being a “self-correcting enterprise” seldom refers to the original researchers but usually to *other* scientists whose pugnacity/irascibility and expertise (both necessary) can give them the energy to investigate claims they find uncongenial or incongruous. Histories of science can provide numerous examples: https://www.amazon.com/Never-Pure-Historical-Struggling-Credibility/dp/0801894212/ref=sr_1_2?dchild=1&qid=1623262855&refinements=p_27%3ASteven+Shapin&s=books&sr=1-2&text=Steven+Shapin

  • Gustavo Barja. Full Proffesor. of Physiology. UCM Univ. Madrid Spain European Union.(World: it should beag should be the goal of at least all non-hypocrit Christians says:

    I agree un general with your article but concernig vitamin D you commit a huge error.

    Correcting immunodeciciency due to deficiency in vitamin D, which is prevalent in 50% of the Spanish (EU) people older than 65 years of age is well known to Spanish Health autirities from dozens of published papers studies performed my spanish medical Drs. But our authorities have done nothing about it!
    Last Autumn 2 medical Drs. performed a pilot study in Cordoba (Spain) with ICU covid-19 patients and tge results were impressive: 35 deaths in tge control group and only 2 deaths in the group supplemented with high dose of vitamin D (100,000 units per day). There are various articles in the Spanish newspapers (e.h. at “ABC”) repirting this. Tge covid-vitamin D book by the vitamin D expert Jeff Bowles (avsilable intermitently ! on Amazon books international) explains this in more detail (pages 248 to around 253; I cite by memory.
    Spanish healt authorities continue to be deaf to this. They only want vaccines and curiously enough they prefer the obviously dangerous mRNA based ones. Concerning this please see:

    “Zhang et al …Young snd Jaenish Whitehead Institute and MIT, CA, MA, PNAS USA (2021), Vol. 118, No 21 e2105968118 Reverse-Transcribed SARS-COV-2 RNA can integrate into the genome of cultured human cells and can be expressed in patient-derived tissues” … amost important timely, excellent, and I would rather declare ELEGANT, paper impossible to refute on sound Scientific and Methodological ground (because the Zhang et al. PNAS study was so well done…)

    Even the famous Science Journal recognized this and then tried hard to criticize ‘discredit- Zhang et al. PNAS USA paper arbitrarily and unscientifically on my opinion through saying that, in their opinion, that integration of SARS COVID mRNA derived short DNA into the genomIc DNA of the infected patients “will be irrelevant”.!! (why so?).

    I do not understand how a scientific journql can maintain just a normal prestige publishing such an unfounded criticism in a case like this affecting more than 1,000 million people? that potentially can very seriously affect the injected people, inducing in them accelerated aging, and thus, cancer, Alzheimer and Parkinson diseases, type II diabetes, cardiovascular diseases, autoimune diseases etc.) at higher incidence in people vaccinated with mRNA based vaccines,,,especially given that 2 other vaccines in the west, one in Russia, and 3 in China are NOT based on viral mRNA but mainly on viral proteins- especially the 3 using attenuated viruses like in the classical and thus more sure to be safe. vaccines from the XXth
    and XXIcenturies …

    When planetary wide vaccinstion is done for the first time in human history, all precautions are few and obvious the risky ones should be discarded by morallly etical persons working as Health and Political Maximum authorities.

    • That PNAS article by Zhang et al.: (https://www.pnas.org/content/118/21/e2105968118.long) deals with actual viral sequences being transcribed, not the mRNA of the vaccine. You realize those are not the same thing, yes? This is (another) reason to be concerned about the virus but NOT the vaccine. The mRNA of the vaccine does not possess the “ancient long interspersed nuclear element (LINE) retrotransposons” possessed by the virus. I’m pretty sure the authors realize this and that is why they do not mention the possibility that you have erroneously hit upon. The mRNA vaccines *cannot* be reverse transcribed into the human genome.

    • Gustavo Barja Full Professor of Physiology, COMPLUTENSE UNIVERSITY OF MADRID (UCM)? SPAIN says:

      Gene Goldbold

      But YOU Have plenty of LINEs providing reverse teanscriptase capabilities and SINE ( both together are 1/3 of human genome).
      See also on IAPs and HERVs and integration of exogenous short nucleic acids in human genome and even possible transmission to offspring
      To all that please read the last pages 63 -68 of the review by Seneff S and Nigh G (MIT, CA, MA) WORSE THAN THE DISEASE? Reviewing Some Possible Unintended Consequences of the mRNA Vaccines Against COVID-19
      Note also that in my country Spain at least the law says that no drug can get into the market unless the ones producing/commercislizing it demonstrated beyond reasonable doubt that they do not cause harm to humans. It is they who should demonstrate that. I do not habmve at all to demonstrate anything.
      mRNA vsccinstion worldwide has never been done before and mid snd long term effects are unknown becsuse phases 1-3 trials are much more short time. I do not understand why this has bern done provided that classical attenuated virus protein based vaccines have been produced and finction well (3 at China, the huge 1,500 million people covid-2 disease free from autumn 2020!
      Then why risk the whole west with these ecperimental mRNA based vaccines. Only due to competition between 4 vaccines at the West?
      Further small DNA fragmentd more easily integrate and we at Spain and Andreas Ivvessa”s group at New York both independently demonstrated in 2010 (Caro et al. Mitochondrion 2010) that mtDNA fragments integration into nuclear DNA is the normal physiological mechanism by which mitochondrial ROS contribute to cause aging.
      Once at the nucleus LINEs csn not discriminate if a smaĺl DNA fragment comes from mitochondria or from retrotrsnscribed at nucleus mRNA vaccine exogenously injected.

      Last and non least if one pland to vaccinate half the population at the West one should be cautious and not use never previously used mRNA vsccines with the potential to integrate into human genes accelersting aging and thus cancer and all the rest of the degenerstive diseases.
      It seems China was so cautious because all their 3 vsccines are protein based classical type attenuated virus ones. And there is no doubt they solved thir pandemics almost one year before us in spite of their huge much higher than our population.

Comments are closed.