Two of the world’s leading medical journals on Tuesday expressed concern about potential flaws in the data produced by a small company to draw major conclusions about Covid-19 — that certain heart drugs are safe, and that the malaria drug hydroxychloroquine is not. The latter finding led to the pause of an important study of hydroxychloroquine by the World Health Organization.
“Serious questions have been raised about the reliability of the findings reported in this paper,” Richard Horton, the editor of the Lancet, where the hydroxychloroquine study was published, wrote on Twitter.
Eric Rubin, the editor of the New England Journal of Medicine, which published the study of heart drug safety, struck a similar tone.
“Substantive concerns have been raised about the quality of the information in that database,” Rubin wrote in the so-called expression of concern published by the journal. “We have asked the authors to provide evidence that the data are reliable.”
The concerns, which have built over the past several days on social media, highlight larger issues with using big databases to draw conclusions about medicines, an approach that has been gaining rigor in the era of big data. Experts warn that conducting such studies properly is far more difficult than it appears.
“This is not for the faint of heart,” said Harlan Krumholz, director of the Center for Outcomes Research and Evaluation at Yale New Haven Hospital. “This is not just a matter of dial-a-study when you get access to data. Well-done studies are based on understanding the provenance of the data and making sure what you are doing is reasonable. There is good science to be done with big databases, but there are also major mistakes to be made. The question is: What happened here?”
Both studies in question used data from Surgisphere, a little-known company based in Chicago that claimed in the Lancet study to have data from 671 hospitals on six continents. The Lancet paper found that the malaria drugs chloroquine and hydroxychloroquine, which had been explored as potential therapies for Covid-19, did not correspond with improved outcomes for patients, and were also associated with higher mortality. The paper in the New England Journal of Medicine reported that blood pressure medications were not associated with worse outcomes in patients with Covid-19. The studies share some of the same authors, including Sapan Desai, who runs Surgisphere.
After suspending the clinical trial arm focused on hydroxychloroquine, the WHO said it would review the data generated so far. That portion of the WHO’s multidrug Solidarity Trial remains paused, delaying answers on a drug that has become a political flashpoint even as the evidence on its potential benefits and risks is murky.
But news reports and experts have raised questions about the integrity of the Surgisphere data. The Guardian, for example, reported on discrepancies in data said to come from Australia in the Lancet study. The Lancet study’s authors have corrected errors about the number of participants, but have stood behind their conclusions.
Outside experts, however, have raised broader concerns. In a May 28 letter to the editor of the Lancet and the study’s authors, more than 180 scientists outlined questions about the statistical analysis and a lack of transparency. The study’s authors did not release the full data underlying the study and did not say which countries or hospitals contributed data, the letter says. The outside scientists called on Surgisphere to at least provide details about how it sourced its data and called for an independent validation of the analysis.
On Tuesday, the group of concerned researchers also wrote to the editor of the New England Journal, raising “similar issues” as they had seen in the Lancet paper. One issue, according to the letter: In some countries, “a relatively small number of hospitals are reported to have provided electronic patient record data to Surgisphere, yet these reports describe a remarkably high proportion of all PCR-confirmed cases in the respective countries.” They flagged specific concerns with data purported to come from the United Kingdom and Turkey.
In a statement, the lead author of the two studies, cardiologist Mandeep Mehra of Brigham and Women’s Hospital, said that the research team “initiated independent reviews of the data used in both papers after learning of the concerns that have been raised about the reliability of the database.” Mehra said the decision for the outside reviews came the remaining co-authors of the papers, “independent of Surgisphere.”
“The goal of the independent third-party auditor is to verify the source data and assess the accuracy of the database and the authors’ findings,” Mehra said.
The Lancet and New England Journal on Tuesday separately issued their expressions of concerns.
In their notice, the Lancet’s editors said the independent audit of the data was “expected very shortly” and noted the “serious scientific questions” that had been raised.
Surgisphere did not immediately respond to a request for comment. In a statement on its website after criticisms were raised about the Lancet study, it highlighted “the validity of our database.” It said its registry was based on electronic health records from customers of its machine learning program and data analytics platform, which allow the company to in turn use data from the records to study “real-world, real-time patient encounters.”
“Together, we stand behind the integrity of our studies and our scientific researchers, clinical partners, and data analysts,” the statement said.
The results of the hydroxychloroquine and chloroquine study in particular grabbed attention, as the drugs’ potential as Covid-19 treatments has become politicized. President Trump and his allies have touted their purported benefits, even as no gold-standard clinical trials have yet produced results and observational studies cast doubt on their efficacy. Trump said he was taking hydroxychloroquine to try to protect himself from the coronavirus.
But there have been concerns raised about the potential side effects of the drugs in people with Covid-19; the Food and Drug Administration, for example, has said it should not be used outside of clinical trials or for patients who are not hospitalized, because of the risk it poses to heart health. Trump’s critics used the Lancet study to argue that by promoting the drug, the president had been endangering the public.
This article looks eerily much like one I just read today, in a professional on-line issue of MedScape …… the writers (Matthew Herper and Andrew Joseph) have failed to state reference to the article from this professional entity. STAT news readers likely read more than you are aware of – you should refrain from any mode of plagiarism !!!
The top journals did not raise the concerns. They finally reflected the concerns of the scientific community, having spent some considerable time deflecting the concerns of the scientific community.
So much fog, can’t trust anything in the American media.
There is a simple study people can do that will show indications of hydroxychloroquine effectiveness – Look at all the people who have been taking it regularly (those with lupus and arthritis who are already taking it), then see the rates of how many came down with Covid, how many had to be hospitalized and how many died.
Compare that data with similar folks in the age group who were not taking the medication. Refine the study as needed to weed out extraneous factors.
You will start to see if there is effectiveness or not.
Has there been a review of Rheumatology patients who have been taking hydroxychloroquine routinely prior to the Covid-19 pandemic and how many of them have contracted the virus?
Are you seriously implying that the “proof” we need that HCQ is in the data of people who are already taking it as a treatment regimen because they are statistically less likely to contract the disease? If so, that’s absurd because there are many other factors and there is no way you could isolate them to make any sound conclusion on correlation/causation.
The conundrum of bias and confounding factors also applies to most drug testing. The objective is to control them to the extent one can and retrospective studies can be of some value. However, given the widespread moderate Zn deficiency in the US, I doubt any effect would be seen.
@Geoffrey, are you seriously implying that it’s invalid to do a retrospective study comparing RA patients taking HCQ with other RA patients, w/ respect to COVID?
If your “absurd” conclusion is correct, then you have just proven too much: NO retrospective study of ANY factor would be valid, because “there is no way you could isolate them to make any sound conclusion.”
OBVIOUSLY, correlation doesn’t imply cause and effect. But if there IS a correlation, it suggests there may be something worth further investigation.
After all, I’m sure you agree: science NEVER “proves” anything to be true. It can only falsify wrong hypotheses. Correct?
@MrPete, Yes, under best circumstances, you can do a clinical trial, but the in the proposed scenario here the control would not meet even the basic threshold for non-bias in any kind of reliable statistical analysis.
Here’s the buried lead in this 821 person study:
“Researchers didn’t find any serious safety problems associated with the drug’s use, including irregular heart rhythms or deaths”
Can we please stop arguing now?
After 65 years and this latest study, can we please allow people to try a [prescribed!] drug no deadlier than aspirin?!
No more useful than aspirin either. VA study found adverse effects, as did studies in Brazil and France.
At best, a bogus nostrum depending entirely on the placebo effect.
Given that at least 98% of the infected survive, you might as well hand out M&Ms and promote them as a remedy that’s “98% effective.”
The VA, which confirms that they give out 42,000 doses of HCQ *per day* completely disowned that “study” by the Gilead grant recipient.
Go ahead, hand them out like M&M’S. At 25¢ a dose, what’s the harm.
Go crusade against something useful.
I am an Ophthalmologist and have seen and specifically evaluated patients for over 35 years on Plaquenil for SLE and Arthritis. Rarely have I heard of any cardiac side effects of the medicine. I suppose all of those patients did not come back because they died, but I doubt that.
It is also a rare side effect that occurs in the retina. In fact, the guidelines for doing visual fields and re-exam were loosened because of the retinal safety of the drug. I personally would instruct my family to approve, or any doctor treating me to prescribe Plaquenil, Zpack and anything else appropriate that would be needed if I were sick with the virus and losing ground. Personally I already take high dosages of Zinc.
I could be wrong (and authors, please correct me if I am) but I suspect that if one checked into the political affiliation of the authors of this article in question, there would be a high statisitical correlation (P=<0.001) to a certain political party.
👍🏻
@andrewnorris – I am with you 100% on both counts – safety and bias!
That’s a great point. Would love to see the statistical significance of this!