Contribute Try STAT+ Today

Two of the world’s leading medical journals on Tuesday expressed concern about potential flaws in the data produced by a small company to draw major conclusions about Covid-19 — that certain heart drugs are safe, and that the malaria drug hydroxychloroquine is not. The latter finding led to the pause of an important study of hydroxychloroquine by the World Health Organization.

“Serious questions have been raised about the reliability of the findings reported in this paper,” Richard Horton, the editor of the Lancet, where the hydroxychloroquine study was published, wrote on Twitter.

Eric Rubin, the editor of the New England Journal of Medicine, which published the study of heart drug safety, struck a similar tone.


“Substantive concerns have been raised about the quality of the information in that database,” Rubin wrote in the so-called expression of concern published by the journal. “We have asked the authors to provide evidence that the data are reliable.”

The concerns, which have built over the past several days on social media, highlight larger issues with using big databases to draw conclusions about medicines, an approach that has been gaining rigor in the era of big data. Experts warn that conducting such studies properly is far more difficult than it appears.


“This is not for the faint of heart,” said Harlan Krumholz, director of the Center for Outcomes Research and Evaluation at Yale New Haven Hospital. “This is not just a matter of dial-a-study when you get access to data. Well-done studies are based on understanding the provenance of the data and making sure what you are doing is reasonable. There is good science to be done with big databases, but there are also major mistakes to be made. The question is: What happened here?”

Both studies in question used data from Surgisphere, a little-known company based in Chicago that claimed in the Lancet study to have data from 671 hospitals on six continents. The Lancet paper found that the malaria drugs chloroquine and hydroxychloroquine, which had been explored as potential therapies for Covid-19, did not correspond with improved outcomes for patients, and were also associated with higher mortality. The paper in the New England Journal of Medicine reported that blood pressure medications were not associated with worse outcomes in patients with Covid-19. The studies share some of the same authors, including Sapan Desai, who runs Surgisphere.

After suspending the clinical trial arm focused on hydroxychloroquine, the WHO said it would review the data generated so far. That portion of the WHO’s multidrug Solidarity Trial remains paused, delaying answers on a drug that has become a political flashpoint even as the evidence on its potential benefits and risks is murky.

But news reports and experts have raised questions about the integrity of the Surgisphere data. The Guardian, for example, reported on discrepancies in data said to come from Australia in the Lancet study. The Lancet study’s authors have corrected errors about the number of participants, but have stood behind their conclusions.

Outside experts, however, have raised broader concerns. In a May 28 letter to the editor of the Lancet and the study’s authors, more than 180 scientists outlined questions about the statistical analysis and a lack of transparency. The study’s authors did not release the full data underlying the study and did not say which countries or hospitals contributed data, the letter says. The outside scientists called on Surgisphere to at least provide details about how it sourced its data and called for an independent validation of the analysis.

On Tuesday, the group of concerned researchers also wrote to the editor of the New England Journal, raising “similar issues” as they had seen in the Lancet paper. One issue, according to the letter: In some countries, “a relatively small number of hospitals are reported to have provided electronic patient record data to Surgisphere, yet these reports describe a remarkably high proportion of all PCR-confirmed cases in the respective countries.” They flagged specific concerns with data purported to come from the United Kingdom and Turkey.

In a statement, the lead author of the two studies, cardiologist Mandeep Mehra of Brigham and Women’s Hospital, said that the research team “initiated independent reviews of the data used in both papers after learning of the concerns that have been raised about the reliability of the database.” Mehra said the decision for the outside reviews came the remaining co-authors of the papers, “independent of Surgisphere.”

“The goal of the independent third-party auditor is to verify the source data and assess the accuracy of the database and the authors’ findings,” Mehra said.

The Lancet and New England Journal on Tuesday separately issued their expressions of concerns.

In their notice, the Lancet’s editors said the independent audit of the data was “expected very shortly” and noted the “serious scientific questions” that had been raised.

Surgisphere did not immediately respond to a request for comment. In a statement on its website after criticisms were raised about the Lancet study, it highlighted “the validity of our database.” It said its registry was based on electronic health records from customers of its machine learning program and data analytics platform, which allow the company to in turn use data from the records to study “real-world, real-time patient encounters.”

“Together, we stand behind the integrity of our studies and our scientific researchers, clinical partners, and data analysts,” the statement said.

The results of the hydroxychloroquine and chloroquine study in particular grabbed attention, as the drugs’ potential as Covid-19 treatments has become politicized. President Trump and his allies have touted their purported benefits, even as no gold-standard clinical trials have yet produced results and observational studies cast doubt on their efficacy. Trump said he was taking hydroxychloroquine to try to protect himself from the coronavirus.

But there have been concerns raised about the potential side effects of the drugs in people with Covid-19; the Food and Drug Administration, for example, has said it should not be used outside of clinical trials or for patients who are not hospitalized, because of the risk it poses to heart health. Trump’s critics used the Lancet study to argue that by promoting the drug, the president had been endangering the public.

  • A randomized trial of hydroxychloroquine was also published in this weeks NEJM showing that it was not effective as prophylaxis in preventing infection in people with known exposure. This is the gold standard study design albeit with caveats regarding how it was determined that the people actually were exposed. The real-word data from databases can also be helpful to build out the picture but of course the data needs to be valid. In the end we need to look to the body of evidence from all sources, academia, industry and government agencies and work together. To date, the data suggests that hydroxychloroquine is not effective. Since there are many other potential treatments to be tested, groups like the WHO performing these huge trials appropriately need to re-evaluate as new data becomes available. It is their ethical responsibility.

    • I do not think you meant Gold Standard. In fact, the report just further raised my concerns and diminishes my view if NEJM and its intent.

      According to NEJM itself, the investigators described it as “a “pragmatic” trial in which participants were recruited through social media and almost all data were reported by the participants.”

      A study based on unverified reports from people recruited via social media has zero validity, subtracts from a serious discussion of this drug, and makes one wonder even more about NEJM.

  • on you tube look up Corona virus update..I think it was March 2nd #33 or #35 gave a some 90% cure rate with the Plaquenil–an FDA approved medicine for 62 years with no cardio-toxicity in the first month of use

    • Cure rate without (hydroxy)chloroquine is at least 95%. So giving it would seem, according to you, to double (at least) the death rate from 5% without the drug to 10% with it. Why is this considered an advance?

  • In 2003 Lancet, infectious disease, vol 3 pp 722 to 728 published a glowing report on Hydrochloroquine. Successfully used with didanosine and hydoxyurea in 3rd world to treat HIV in place of expensive Proteases.

  • Thank you for bringing this out into the open. If the general public would just open their eyes and see that anything that comes out of President Trump’s mouth must be made to be a farse and ridiculed. What bothers me is that all of us that rely on “evidenced based medicine” were let down big time. Who can you believe anymore?? Politics needs to stay out of medicine….big Pharma needs sent away….there should be no place for lobbiests in this arena.
    Just my thoughts for what it’s worth.

  • I am a retired Physician and I have listened to the cardiologists on CNN and MSNBC especially scaring the public about Hydroxychloroquin. If I were to test positive I woud want HCQ, Azithromycin, and Zinc started at once. if I am sick enough to be hospitalized, the drugs might all do harm or show no benefit. But the postulated mechanisms of the drugs suggests why they should be used early if they are to work. HCQ is used in auto -immune diseases to tamp down the immune system. Zinc is known to have activity against the “cold viruses” and is used in coldeze. HCQ may well facilitate the entry of Zinc into the cells. Azithromycin I believe has shown some activity against some viruses in vitro as has Hydroxychloroquin. The HCQ and Azithromycin are bothgeneric. Testing the above drugs on hospitalized patients are trials that are purposely designed to fail. Big Pharma can make no money off the generic drugs above listed. Every Physician I know wants the 3 drug combination for themselves if they can get it adminsitered early. And the media is oblivious to everything I have just said. My general understanding is that in at least some patients more damage is caused by the immune system response (especially the cytokine storm)and in addition to possible direct anti-viral activity the 3 drug combination listed above might well blunt the cytokine storm. Were I Sherlock Holmes addressing Dr. Watson to explain my “conclusions” (hypotheses) stated above I would say It’s elementary my dear Watson!

    • This seems to be fair analysis, but it seems that you are talking about two different things. The important part of what you raise is how you precursor your second sentence- “if I were to test positive”. You also mention the application of this regimen if you were to be hospitalized. Yes, the mainstream media, particularly CNN and MSNBC pointed out this original publication in the Lancet that is now in question, but the main point that they were “especially scaring the public” as you say was in regard to self administration of HCQ as a preventative like what Trump was suggesting. This is the key differentiation. Now you have right leaning outlets using the backtracking and questioning of the Lancet to try to vindicate Trump. This is what should be scary. Let’s just be clear under what circumstances the drug could or should be taken and about its application as a preventative.

    • Sir, I think you have struck to the pith of the matter. Add the odd coincidence that Zn and D3 deficiency prevalence is highest in the elderly, African Americans, and Hispanics and the picture is almost complete. Finally, we might want to look at the effect of low melatonin levels (also prevalent in the elderly) on cytokine storms.

    • “Geoffrey” to kindly cite where Trump advised people to self medicate without a valid prescription. I need the clip of him stating this, please. NOT Vox or CNN’s liberal based interpretation of “what he must mean”

    • @MS, Trump sat in front of a camera and said himself that he was taking it as a preventative. As a result, there are documented cases of people self administering, which was clearly a foreseen outcome or consequence. He does not appreciate the weight of his words.

    • @Geoffrey,
      President Trump’s comments, in context, urged people to follow their doctor’s advice. He was urging the Powers That Be to investigate, to make this available as an *option.* And that is what was done. It was approved on a special basis requiring reporting results back to CDC!

      It is only when the media (which you apparently trust) take his statements out of context, that people come to believe something false.

    • @MrPete, Firstly, Trump never “urged” anything. Secondly, the CDC has not approved HCQ as a preventative in non-Covid-19 positive patients. This is the context in which Trump was discussing its use and this is the problem. I am not relying on the media, I watched his Q&A session live and again on replay. These are his implications, not the media’s. I suggest you go back and watch the video and if you still don’t get it, then perhaps a little elementary comprehension exercises are in order. Then again, you probably don’t believe that peaceful protestors were gassed and shot with rubber bullets because Trump is saying that they weren’t so it might be a lost cause.

  • Hydroxichloriquine has been in constant use for over 9O years. Varius ailments are being treated with HCQ, and it is considered a very safe medicine. There are two objections to its use–it is cheap and has no patent. Big Pharma, Gates want a mandatory vaccine and Gates wants to inject a chip with the medicine. Those who refuse will not be able to travel or work. Megalomaniac s can never have enough money, power or control. Gates is also the WHO largest doner.

    • What are you talking about? Yes, it is a useful drug in the right application. You don’t just take it when there is no clear benefit for ailment in question, particularly when side effects vis-a-vis that ailment and/or in combination with other treatments are unknown. Come on….

    • Is this a Bernie Madoff situation? If so, is this a one off situation? It seems that the Lancet articles prestigious authors including a top cardiologist from Geneva bear a significant amount of responsibility here not just Dr. Desai, as does the Lancet. It will be very important to thoroughly and fairly assess the concerns raised by Dr. Watson and the cosignatories. The phony elevation of “big data” as the superior way to gain medical knowledge is part of the dumbing down and disempowerment of the medical profession. Given the challenges and complexity inherent in medical research, translation and practice the substitution of “big data” entrepreneurs supported by so called highly credentialed and highly paid experts is very troubling. My view is that when it comes to what passes the grade at the Lancet, this is the tip of the iceberg of a huge problem such as the Long-standing misuse of experts by Pharma ignored by the Lancet and now possibly by big data bunksters.

  • You know they’re both to do over causing heart problems I find outrageous. I have asd. Left side of heart enlarged had open heart surgery in 1994 to repair hole. Have had severe RA ever since.I have taken this drug and guess what not problems line thisnline. Didnt find it effective for the RA for which its been widely used. They just cant make huge amounts of money for it from what I see.

  • Need reliable data. It concerns me that these articles got published in the first place if reliability of the data was so questionable. Whatever the truth about the value of the therapy our journals have let us down.

  • HCQ is used for several decades for Malaria. ?this threat was not present. Presently using for few SLE patients. ?this threat was not present. Whether It works or not is different but why this alarming side effects now in COVID 19

  • These studies were terrible. Why did theses editors of Lancet and NEJM not demand more rigor before publishing such trash?

    • NEJM and Lancet published the junk studies ’cause it’s Covid. Anything about Covid can get published.

Comments are closed.

Your daily dose of news in health and medicine

Privacy Policy