Last Wednesday, Gilead announced that its drug, remdesivir, sped the time it took patients to recover from Covid-19. Full data from the study were released by the National Institute of Allergy and Infectious Diseases in a statement at 1 p.m. By 3:35 p.m., Walid Gellad, director of the Center for Pharmaceutical Policy and Prescribing at the University of Pittsburgh, had noticed a problem.
Was there any explanation, he asked in a tweet, for why the study’s main goal had been changed just weeks before? He included a screenshot of a record of the change on the U.S. government’s online registry of clinical trials.
It’s a big concern — and hypothetically could have affected how trial results were interpreted. It could be all the more concerning because the Food and Drug Administration used the data as the basis to authorize emergency use of remdesivir.
But new comments from the NIAID, made to STAT, may make the issue less fraught.
In clinical research, the issue of what is designated as the “primary endpoint” is not merely academic. In medical studies, as in pool, you have to call your shot before you sink it. The reason is the same: this lowers the odds that there will be a positive result that occurs only by chance. You don’t get credit for accidentally putting the ball in the corner pocket when you meant to put it in the side; researchers don’t get credit for finding a positive result they could not predict ahead of time.
Changing a primary endpoint when you already know the data is even worse. It can even amount to outright research fraud. To mix sports metaphors, researchers often liken it to drawing a target around an arrow after you shot it, and then claiming a bull’s-eye.
In this case, it appears not to have happened. The NIAID said Thursday that the change was made while still blinded to all outcome data because agency statisticians had performed modeling showing that the original endpoint — an eight-point scale of how subjects were doing that ranged from dead (the worst outcome) to out of the hospital with no restriction on activities and no need for oxygen (the best) on the 15th day of the study — might not detect a difference where one existed. Instead, they changed the goal to the time it took patients to no longer require supplemental oxygen in the hospital or to be out of the hospital entirely.
In an interview, H. Clifford Lane, the clinical director at the NIAID, reiterated that the researchers made the change before any data were revealed to them. The decision to make the change, he said in a follow-up email, had been made during a call on March 22, when only 77 patients were enrolled in the study — long before researchers could have seen data. The change had to be approved by the researchers, Gilead, and the Food and Drug Administration; this was done by April 2, Lane said.
But Lane also said that the change apparently didn’t make any difference. Because if they had stuck the study’s original primary endpoint, the study still would have been positive.
“The first secondary outcome, which is the old primary endpoint, the prior primary endpoint, has a highly statistically significant difference,” Lane said, adding that he “feels badly” that this has not been made public previously and adding that the data are still being “cleaned up,” part of the normal checking process for data in clinical studies. The p value, a number used to determine if a result is statistically significant, was similar to the one on the new primary endpoint: 0.001. The cutoff for statistical significance — to determine whether a result “counts” and is not just a matter of chance — is usually less than 0.05.
If that’s so, it’s still possible to have worries about how the endpoint was changed, and there could be other potential concerns when the full data are released. But it does mean the endpoint change is less of a concern than it might at first appear — although, again, many researchers will want to wait for a full data presentation.
Said Gellad: “If the NIH is telling you the original endpoint was also clearly in favor of remdesivir, then the endpoint change is clearly less of an issue.”
About the Author Reprints
Great reporting, Matt— on a somewhat arcane, but crucial, aspect of clinical trial integrity. Your “pool”/(billiards) and “darts” analogies were great! It would be interesting to know what motivated this new modeling, and the sudden epiphany that the numeric ratings scale wouldn’t work (although it did…) Ideally, there would be a solid chain of logic unrelated to the trial itself. “Peeking” at the data and making design adjustments typically follows a prespecified “interim analysis” and involves a statistical penalty for the same reasons that you mention. That does not appear to be what happened here. Bottom line, I guess: It doesn’t “look good”, but it’s not the worst thing either, given the explanation of the circumstances.
I don’t have a problem with them changing the endpoint, particularly since this was done well in advance of unblinding, and especially since the new endpoint is still something useful: time to hospital discharge. If they’d picked, for example, viral load, I would have deemed this study essentially worthless from a clinical perspective. But the data as presented – shorter hospital stay, trending improvement in mortality – are a decent ‘base hit,’ and base hits often win games.
Dr Fauci seemed excited. That’s enough for me.
Better than some fool talking
Lets not use this drug. It raises liver enzymes and has a possibility of liver failure . Hydrochloriqune seems like a better choice . Lets not worry about making money on a new drug for once and pick the best choice instead.