Editor’s note: The day after this First Opinion was published, STAT received by email a letter written by by Peter Marks, director of the FDA’s Center for Biologics Evaluation and Research, disputing the fact that the FDA used only one reviewer for each of three parts of its review of the Pfizer/BioNTech vaccine, as Peter Doshi and Matthew Herder suggest here. You can read the full FDA letter and the authors’ response to it here.
In what is arguably the most important decision the Food and Drug Administration has made this year — its emergency use authorization of the Pfizer/BioNTech Covid-19 vaccine — the agency apparently assigned only a single reviewer in each of two key scientific disciplines (clinical and statistics) to do the work in three weeks that usually takes months to do.
The FDA’s authorization last week followed similar authorizations in the United Kingdom and Canada. But the FDA’s decision is particularly important because of its reputation for being the international “gold standard” in regulatory rigor.
Unlike its counterparts in other countries, the FDA is believed to be the only drug regulator in the world that consistently receives and reviews patient-level data from the clinical trials that underpin drug and vaccine approvals. To perform such rigorous analyses, the FDA typically spends around 10 months (a mere six months for applications given “priority review” designation) in an effort that involves reviews by experts representing various scientific disciplines: clinical medicine, statistics, pharmacology, chemistry, pharmacovigilance, and more. Together, these reviews form an “action package” which, by law, must be made publicly available 30 days after approval.
Given the urgency of the pandemic, the review of the Pfizer/BioNTech vaccine was conducted far faster than usual. The centerpiece of the analysis was data from the company’s 44,000-participant Phase 3 trial. FDA reviewers had just three weeks, from Nov. 20 to Dec. 11, to complete their analyses. It was a monumental task, which raises the question: Why didn’t the FDA devote additional reviewers to it? According to the FDA’s review memo, some scientific disciplines, such as pharmacovigilance, had multiple reviewers involved. But the two disciplines tasked with examining the clinical trial data and results, the clinical and statistical reviewers, were seemingly left to do their work solo. (Editor’s note: The FDA disputes this assertion in the letter below.)
This seems wholly inadequate on at least two levels. First, without additional reviewers it is hard to comprehend how the work of several months could be squeezed into a matter of 22 days (including Saturdays and Sundays). In-depth review calls for examining patient-level data — a large feat that involves auditing and reviewing individual case records as well as independently rerunning analyses on the raw data.
Before the pandemic, it was typical to see just one clinical reviewer’s name for any given application. But given the stakes — and the time crunch — involved with reviewing Covid-19 vaccines, we would have thought the agency would do an even more thorough job than normal. But that does not appear to be the case.
One of us (P.D.) raised questions about potential unblinding in the trials through the vaccine’s side effects, as well as about the confounding effects of fever- and pain-reducing medications, which participants in the vaccine arm took three to four times more often than those in the placebo arm. Yet the FDA’s review shows no evidence that any of its scientists investigated either of these issues, and without more scientific staff devoted to the task it is hard to imagine how they could.
As one of us (M.H.) has investigated in the past with respect to FDA approvals not related to Covid-19, disagreements within the agency about whether to approve an intervention or set limitations upon its use are relatively common during the review process. Among 174 approvals examined between 2011 and 2015, 42 (24%) contained at least one disagreement among reviewers.
A key takeaway from this research is that a real strength of the FDA is its capacity to entertain dissent with a view to making better judgments about complex scientific evidence.
Such differing opinions were on display at last week’s FDA advisory committee meeting, where four advisers voted against the emergency use authorization that the FDA ultimately granted the next day. But that was the FDA’s external advisory committee.
Whether such disagreement also exists within the FDA is unclear. By seemingly assigning only one clinical, one statistical, and one toxicology scientist to review the Pfizer/BioNTech Covid-19 vaccine, it seems that discussion, let alone disagreement, was curtailed by design. The pressure on those lone FDA reviewers to do their work in record time, and do it without raising serious questions about the data, was likely immense.
Having an official second or third reviewer for all core scientific disciplines would have helped. Not making more reviewers available strikes us as an effective way to prevent in-depth assessment of the underlying data and limit the possibility of dissent. Shortcuts in the regulatory process undermine the very purpose of regulation to protect the public. If the goal was speed at all costs, we should just get rid of regulators. Otherwise, they need all the resources they can get to do their job, including the fast-approaching decision about whether to authorize Moderna’s Covid-19 vaccine.
Peter Doshi leads the Restoring Invisible and Abandoned Trials initiative and is an associate professor of pharmaceutical health services research at the University of Maryland School of Pharmacy. Matthew Herder is director of the Health Law Institute at the Schulich School of Law and an associate professor of pharmacology in the faculty of medicine, both at Dalhousie University in Canada.
Competing interests: Doshi has received grants from the FDA, and the Laura and John Arnold Foundation, and is an unpaid member of the Reagan-Udall Foundation for the FDA. Herder is a member of the Patented Medicine Prices Review Board, Canada’s national drug price regulator, and holds grants from the Canadian Institutes of Health Research.
To the Editor: The December 17 opinion from Peter Doshi and Matthew Herder is inaccurate and mischaracterizes the work of FDA career scientific staff involved in the review of the request for emergency use authorization (EUA) for the Pfizer-BioNTech Covid-19 vaccine. The authors’ opinion does not demonstrate an understanding of FDA’s team review processes that went into review of the EUA request.
Agency staff have been working around the clock for months, including nights, weekends, and holidays — long before the EUA request was submitted — providing feedback and advice to all sponsors developing Covid-19 vaccines. The authors cite FDA’s clinical review memo regarding the Pfizer-BioNTech Covid-19 EUA as evidence that more staff should have been assigned to conduct reviews. In fact, more than one hundred staff — including senior management from across FDA’s Center for Biologics Evaluation and Research, and FDA as a whole — contributed to this effort.
The authors clearly fail to understand that this is not the equivalent of an academic manuscript. The individuals listed are simply the leads for each review discipline, and do not represent the entire team of dedicated scientific review staff who worked tirelessly to provide thorough review of the EUA request.
The effort put forth by FDA staff to complete its review and authorize Pfizer-BioNTech’s Covid-19 vaccine for emergency use in 22 days following formal submission of the EUA request was heroic. This was not business as usual. FDA undertook an all-hands-on-deck approach to this work.
Because of the suffering caused by this pandemic, our career scientific review staff felt the responsibility to work through the review process with a tremendous sense of urgency while carefully doing their jobs to ensure that any authorized vaccine meets our rigorous scientific standards that Americans — and the world — have come to expect. To suggest otherwise is an affront to their incredible effort.
Peter Marks
Director, Center for Biologics Evaluation and Research
Food and Drug Administration
The authors respond: We appreciate Peter Marks’ response to our commentary raising questions about the rigor of the FDA’s review process for the Pfizer/BioNTech vaccine emergency use authorization. Marks faults us for taking at face value what is written in the agency’s review memo, which lists just one medical reviewer and one statistical reviewer as being involved in the review, which had a 22-day timeline from receipt of the application on Nov. 20 to the FDA’s decision on Dec. 11.
Marks says that the FDA staff has been working for months. We believe that. But it misses our concern that review of the results of the Phase 3 trial — which was completed just weeks ago and was submitted to the FDA on Nov. 20 — means the trial data were reviewed in three weeks, lightning speed compared to FDA’s normal months-long process. Marks provides no examples of how the underlying patient-level data were critically analyzed during that 22-day period. As our commentary suggests, we believe that a thorough examination of these data is not feasible within that time frame, at least not by a single clinical and statistical reviewer.
Regarding the depth of the FDA’s review, Marks says that more than 100 staffers “contributed to the effort” and that the individual names listed in the “Review Team” are “simply the leads for each review discipline.” We believe this, too, but wonder why two individuals are listed for pharmacovigilance and three for chemistry, manufacturing, and controls (CMC), but only one each is listed for clinical, biostatistics, and toxicology?
More importantly, as we explained, our concern is not simply about the quantity of reviewers — it is about quality of the process. We saw no sign that the agency assessed the possible impact of potential unblinding in the trial given the side-effects of the vaccine. And we wondered how many people were officially tasked with the responsibility of forming an independent view of the science. We remain concerned that there were no clear structural mechanisms by which reviewers who might have held dissenting views about the strengths and limitations of the application could freely document and discuss their concerns.
We are heartened to hear that the agency mustered “an all-hands-on-deck approach” to this review. It remains curious to us as to why the memo, combining all disciplines’ reviews, totals just 57 pages — much smaller than many solely clinical reviews. Ensuring that the memo accurately reflects all the work that went into the review, especially how and what was analyzed during the 22-day span, and lists all those involved in the effort is, in our estimation, the best course to building trust in the agency’s decision-making.
Peter Doshi and Matthew Herder
I completely agree, and I totally blame Trump. This idiot has done more to harm our institutions than I could have possibly imagined. He has made every attempt to politicize, make money, or inflate his ego at every opportunity afforded to him, and his party in Congress almost completely backs him up. I can’t believe actually having a functioning and respectable government isn’t important to his base, but here we are…
Wise Words from C.S. Lewis in 1948
The following from C. S. Lewis. was on my fraternity email list this morning. It was written in 1948 after the dawn of the atomic age.
In one way we think a great deal too much of the atomic bomb. “How are we to live in an atomic age?” I am tempted to reply: “Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.”
In other words, do not let us begin by exaggerating the novelty of our situation. Believe me, dear sir or madam, you and all whom you love were already sentenced to death before the atomic bomb was invented: and quite a high percentage of us were going to die in unpleasant ways. We had, indeed, one very great advantage over our ancestors—anesthetics; but we have that still. It is perfectly ridiculous to go about whimpering and drawing long faces because the scientists have added one more chance of painful and premature death to a world which already bristled with such chances and in which death itself was not a chance at all, but a certainty.
This is the first point to be made: and the first action to be taken is to pull ourselves together. If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things—praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.
— “On Living in an Atomic Age” (1948) in Present Concerns: Journalistic Essays