Contribute Try STAT+ Today

Last Wednesday, Gilead announced that its drug, remdesivir, sped the time it took patients to recover from Covid-19. Full data from the study were released by the National Institute of Allergy and Infectious Diseases in a statement at 1 p.m. By 3:35 p.m., Walid Gellad, director of the Center for Pharmaceutical Policy and Prescribing at the University of Pittsburgh, had noticed a problem.

Was there any explanation, he asked in a tweet, for why the study’s main goal had been changed just weeks before? He included a screenshot of a record of the change on the U.S. government’s online registry of clinical trials.

It’s a big concern — and hypothetically could have affected how trial results were interpreted. It could be all the more concerning because the Food and Drug Administration used the data as the basis to authorize emergency use of remdesivir.

advertisement

But new comments from the NIAID, made to STAT, may make the issue less fraught.

In clinical research, the issue of what is designated as the “primary endpoint” is not merely academic. In medical studies, as in pool, you have to call your shot before you sink it. The reason is the same: this lowers the odds that there will be a positive result that occurs only by chance. You don’t get credit for accidentally putting the ball in the corner pocket when you meant to put it in the side; researchers don’t get credit for finding a positive result they could not predict ahead of time.

advertisement

Changing a primary endpoint when you already know the data is even worse. It can even amount to outright research fraud. To mix sports metaphors, researchers often liken it to drawing a target around an arrow after you shot it, and then claiming a bull’s-eye.

In this case, it appears not to have happened. The NIAID said Thursday that the change was made while still blinded to all outcome data because agency statisticians had performed modeling showing that the original endpoint — an eight-point scale of how subjects were doing that ranged from dead (the worst outcome) to out of the hospital with no restriction on activities and no need for oxygen (the best) on the 15th day of the study — might not detect a difference where one existed. Instead, they changed the goal to the time it took patients to no longer require supplemental oxygen in the hospital or to be out of the hospital entirely.

In an interview, H. Clifford Lane, the clinical director at the NIAID, reiterated that the researchers made the change before any data were revealed to them. The decision to make the change, he said in a follow-up email, had been made during a call on March 22, when only 77 patients were enrolled in the study — long before researchers could have seen data. The change had to be approved by the researchers, Gilead, and the Food and Drug Administration; this was done by April 2, Lane said.

But Lane also said that the change apparently didn’t make any difference. Because if they had stuck the study’s original primary endpoint, the study still would have been positive.

“The first secondary outcome, which is the old primary endpoint, the prior primary endpoint, has a highly statistically significant difference,” Lane said, adding that he “feels badly” that this has not been made public previously and adding that the data are still being “cleaned up,” part of the normal checking process for data in clinical studies. The p value, a number used to determine if a result is statistically significant, was similar to the one on the new primary endpoint: 0.001. The cutoff for statistical significance — to determine whether a result “counts” and is not just a matter of chance — is usually less than 0.05.

If that’s so, it’s still possible to have worries about how the endpoint was changed, and there could be other potential concerns when the full data are released. But it does mean the endpoint change is less of a concern than it might at first appear — although, again, many researchers will want to wait for a full data presentation. 

Said Gellad: “If the NIH is telling you the original endpoint was also clearly in favor of remdesivir, then the endpoint change is clearly less of an issue.”

  • That’s a poorly constructed description of what this medicine does. I can’t understand what is being said.

  • Hidden results! If it is such a clear benefit why can’t we see all the data (like CPR results – positive to negative conversion, patient characteristics, oxygen saturation levels, other interventions, heterogeneity in outcome between different hospitals, etc). Considering the Hubei trial came out negative, as usual with Gilead, once again they are in murky grounds including their own strange results from 5d v. 10d trial (is 1d as good as 5d?) and their rhesus model (why not include interventions at 12h, 24h, 48h after the inoculation challenge?).

  • So what are the results? Remdisivir reduces days in hospital by how much in what kinds of people? Were severity of illness, age, gender etc. controlled for? And the trend in decreased mortality was how much? and when they left hospital how many days at home until recovery? vs a matched group that did receive the drug?

  • Hello Mr Herper,

    I hope you’re well🙏🏾

    I’m not a medical professional, well just with my children who are now grown, scientist or non of that and I’ve been keeping up with news, articles, etc., but let me know if I’m understanding this at least a little.

    From what you have written, this “Remdisivir” shortens the recovery time by some days.
    Those days in between from the first projection to the recent one….could still require more recovery time, just not in the hospital, vs being fully recovered and be discharged with no trace of the virus? But because of the rush to “find something…anything” that half way work, will be projected as a breakthrough although people will leave the hospital not 100% but well enough for the data to show successful?

    Please excuse me if I’m sounding a little off but I’m just trying to figure this all out. What have I got to lose?

    Thank you and be safe🙏🏾

    Sebrena

    • The trial was blinded. Even if they’d let some patients out of the hospital before they were fully recovered, there would have been no way to predict they were letting the ‘right’ patients out of the hospital. The design – blinded – makes it very difficult to influence the outcome in the way you’re suggesting. Also, this drug is not really a ‘breakthrough’ or miracle cure or anything else. It works a little bit to shorten hospital stays. Most people would prefer a shorter hospital stay, and the shorter stay suggests the patients are recovering faster. But it’s hardly a game changer. Still, this result could be particularly useful in the case that hospital space becomes extremely limited, e.g. if a second wave in the fall is worse than this first wave.

  • “You don’t get credit for accidentally putting the ball in the corner pocket when you meant to put it in the side; researchers don’t get credit for finding a positive result they could not predict ahead of time.”

    Are you saying that Alexander Fleming’s accidental discovery and isolation of penicillin should receive no credit? Such unexpected observations, both large and small, are commonplace. I accidentally cloned the gene for the nuclear envelope lamin B in the late 80s, it led to one of my better papers.

    • If you’re going to make a statistical argument that your treatment works, you don’t get credit for unexpected positive outcomes. If you get an unexpected positive result, you can design a new trial to determine whether it’s a real effect.

    • That does not address the question I asked, or the quotation from the article.

  • C’mon Big Pharma, stop pushing your “pigs” as prom queens.

    We’re not fooled by your use of the dark arts and public relations wizardry…

  • As a layman, maybe I do not understand this article, but it seems like the original endpoints were what everyone cared about. Whether the disease is going to kill me or not. For that matter, even the original endpoints seem a bit incomplete, as the best outcome was the patient out of the hospital and breathing on their own, but some of those patients will have various long term health problems, if what is reported is correct.

    After seeing all the big enthusiasm for the drug, and all the talk of reopening, one has the fear reopening will be partly predicated, even if this is not directly claimed, on the thinking “What’s the big deal, we have a drug which works on this now?” – when that seems to be a big exaggeration.

    Reading about the trials, I saw the claim that the observed 30% lower death rate was not considered proven. I am not able to assess the math to verify that, but that, and long term health damage, are the measures I care about.

    Right now, I can not tell if I should feel excited and relieved that it is safe to reopen the country, which, without any doubt, will result in a vastly greater number of new infections, or dread, because we are racing into a disaster.

    The only thing I feel certain of is that good news, which is what this remdesivir study was presented as, is very dangerous, unless it holds up.

    • The reason that the reduction in death rate did not reach statistical significance is that death is a relatively rare event, therefore the sample size (number of patients in the study) would have had to be much larger in order to reach statistical significance That is why it was reported as a “trend.” It isn’t that it “isn’t proven” it’s that it CAN’T be proven – with statistical significance — with this number of patients in the study. Actually, this is a pretty large sample in general which is why they were able to conclude that length of illness was statistically significant at a p=0.01 (that means there is less than one percent chance that the results could have occurred by chance — or — it is 99% positive that the difference WAS attributable to the drug). Most such trials look for a p-value of 0.05 (a 5% chance that the result could have happened by chance) which suggests that the difference in length of illness is not JUST significant, it is VERY significant.

    • Liz – thank you for this important information. I have some awareness of the meaning of p value but did not know they are generally looking for a p value of .05 or that the problem in the study was not enough fatal outcomes.
      So, now the enthusiasm makes more sense to me. But, I am not clear on the availability of this drug, nor when studies will show just exactly how much it can help.

  • It’s amazing how who said the drug didn’t work and even posted the Chinese fake study in their website. Now they’re asking gilead for the drug

    Why should anyone believe in the Chinese or who for what matter?

  • Yeah, if you totally don’t understand how the regulatory structure works and how all major changes like this have to go through full review IRB review for every site running the trial, (I’ll let you Google who is in an IRB and what they do), someone might be concerned, but journalists have shown time and time again they have no clue when it comes to reporting on scientific advancement/discovery.

Comments are closed.