Failure to warn: An early warning system for drug risks falls flat

This is the second of two stories on monitoring the safety of new drugs — read the first here. 

WASHINGTON — In 2007, after a number of dangerous medicines were pulled off the market, Congress ordered the Food and Drug Administration to set up an early warning system to detect other harmful drugs before more people died.

The FDA responded by creating a system called the Sentinel Initiative, which mines insurance data and medical records to identify possible risks.

But a system that was touted as a revolutionary way to glean valuable information from electronic health records and protect patients has had little measurable impact, according to a STAT examination and interviews with drug safety experts. Ten years and $207 million later, critics question whether Sentinel can adequately identify risks involving drugs that are already in consumers’ medicine cabinets.

To date, its records have been used to revise only two drugs’ warning labels, which provide key information on side effects and other possible “adverse events” associated with medications.

When officials announced plans for Sentinel, they did so with great fanfare. The secretary of Health and Human Services at the time, Michael Leavitt, said it would be a “quantum leap forward in FDA’s capacity to monitor medical products that are already on the market.”

It “was going to transform how we monitor drugs after they are approved,” said Thomas J. Moore, a senior scientist at the nonprofit Institute for Safe Medication Practices. “Eight years later, it’s valuable, but much less so than originally advertised.”

Critics point out Sentinel lacks important data: It relies primarily on insurance company billing records that have limited information; patient records include few people over age 65; the medical providers that feed data into the system use different diagnostic codes for the same conditions; and, perhaps most important, the vast majority of records for people who died as the result of medications never makes it into the system.

Dr. Joseph Ross, an assistant professor of medicine and public health at Yale University and an expert in new drug safety, said he believes it’s impossible to know if Sentinel actually detects the adverse reactions it is hunting for.

“The truth of the matter is, we don’t ever really know how well Sentinel is working,” said Ross. “We don’t know if there is a safety problem that Sentinel is picking up or not.”

FDA officials and others involved in Sentinel defend the progress made so far and stress its potential. Dr. Richard Platt, the principal investigator of the project, based at the Harvard Pilgrim Health Care Institute, said Sentinel has already been valuable in helping confirm the safety of certain medications so they could be kept on the market.

Platt also noted that Sentinel can be used to access 193 million patient records and compare side effects between two drugs used for the same condition, an extremely useful approach when trying to identify possible risks associated with medications.

“There’s an increasingly sophisticated capability to ask certain kinds of questions and get reliable answers in a relatively brief period of time,” Platt said.

The importance of developing a system to better detect risks after drugs hit the market has never been in doubt.

For years, the FDA has acknowledged that its adverse event reporting system, called FAERS, is terribly incomplete. Drug companies are required to report deaths and serious problems believed to have been caused by their products to the FDA, but they rely on doctors and patients to alert them — and studies show that few bother. (Doctors and patients may also report adverse events directly to the agency, but most do not.) The FDA also lacks the staff needed to follow up on these reports to determine if they are accurate.

“Even the FDA will tell you, FAERS is at best the tip of the iceberg,” said Ross. “And at worst it breeds false confidence, because there is rampant underreporting.”

At a time when the FDA wants to get drugs on the market ever faster, and when the agency is considering increasing reliance on “real-world evidence,” critics say Sentinel’s limitations are troubling.

“It’s always been very hyped as some grand system that will help us detect safety issues sooner,” said Dr. Michael Carome, a former Health and Human Services official who runs Public Citizen’s Health Research Group. “But they’ve poured in an extraordinary amount of investment and it’s not clear that we’ve gotten a good return on the taxpayer expenditure.”

Although manufacturers generally must submit results of clinical trials before the FDA approves a drug, these can be small, and don’t always reflect the danger to the general public. To assess that risk, the agency waits for what it calls post-market surveillance.

But sometimes those risks don’t become apparent soon enough. A decade ago, a series of deaths and serious damage from Vioxx, a painkiller, and Ketek, marketed as the first of a new class of antimicrobial agents, led Congress to investigate the FDA’s system for monitoring the safety of new drugs.

Vioxx (rofecoxib), made by Merck, had caused an estimated 88,000 heart attacks and 38,000 deaths, according to the journal the Lancet. Ketek (telithromycin), linked to severe liver damage, was said by congressional investigators to have been approved based on fraudulent safety data.

In a series of combative congressional hearings in 2007, lawmakers grilled FDA officials on how they missed the signals of such a disaster and how they would avoid another one. The officials acknowledged that underreporting of serious adverse events was a huge problem.

In response, Congress ordered the FDA to develop a new system of post-market surveillance of drugs and biological products such as vaccines.

Two years later, the FDA launched its a pilot program for Sentinel to test the feasibility of plans to use electronic health records from insurers and health maintenance organizations. Not until 2016 did Sentinel move out of its pilot phase and begin operating on a larger scale.

Today, Sentinel includes data from a group of 19 insurance companies, HMOs, and hospitals. Those entities include Vanderbilt University Medical Center, the HMO Research Center, Aetna Inc., Kaiser Permanente Center for Effectiveness and Safety Research, and others.

The overall process behind the program is straightforward: If FDA officials notice reports in the FAERS system about bad reactions to a drug or become worried about possible risks based on research studies or other information, the agency alerts Harvard Pilgrim and requests a closer review. Platt said Harvard Pilgrim’s first step is figuring out if any of the partners in the Sentinel system can actually answer the question.

If so, Harvard Pilgrim selects one or more of those partners to review their records. The safety history of the new drug in question is then compared with an older drug used to treat the same condition. No patient names or other identifying data are used.

“We identify people who started taking the drug, and ask how often do they experience whatever the problem is that the FDA is concerned about,” said Platt. “The question is, is this an unusual frequency of adverse events?”

Sentinel can also review patients’ earlier drug history.

To Platt, Sentinel complements the FAERS system, by permitting researchers to compare drugs to see which one causes more adverse reactions — something that can’t be done with the FAERS spontaneous reporting system. That kind of analysis was possible before, but only with the benefit of lengthy clinical trials.

“Before, it might have taken a couple of years and been very expensive. Waiting a long time to get an answer is potentially risky,” Platt said. “What Sentinel has allowed FDA to do is to get an answer, often within weeks, or if extremely urgent, less than a week.”

So far, Sentinel has run hundreds of queries. But, according to both Harvard Pilgrim and the FDA, its work has contributed to label changes for only olmesartan medoxomil, which is used to treat high blood pressure, and RotaTeq, a liquid oral vaccine for use in infants and children to prevent gastroenteritis.

Olmesartan’s label was revised to warn patients of possible intestinal problems. Prescribing information for RotaTeq was changed after Sentinel noted an increased risk of intussusception, a medical emergency in which the bowel is obstructed.

Dr. Robert Ball, deputy director of FDA’s Office of Surveillance and Epidemiology, which oversees Sentinel, acknowledged the program’s limited impact to date. But he said that Sentinel has only been working at full capacity for one year, following its pilot phase, and will prove increasingly valuable over time.

“It’s taken a while to build the system,” he said. “We expect there will be more situations where the data from Sentinel will be used toward regulatory decision making.”

Ball also acknowledged limits on available data. Sentinel doesn’t include information on over-the-counter medications, for example, or on variables such a patient’s body mass index or whether he or she smokes, information that would help evaluate whether a drug is causing certain reactions.

An interim assessment of the system produced for the FDA by outside consultants, whom the FDA would not identify, noted its accomplishments but also highlighted its limitations.

The report, released in fall of 2015, said that Sentinel’s reliance on insurance claims data means there is no detailed information that is typically included in a patient’s medical history. For example, there are no laboratory results; and while there is information showing what was prescribed, there’s no data, unlike in a clinical trial, that show what was actually taken. Some patients may not always take their medicines as prescribed.

The auditors, who interviewed FDA staff and “stakeholders,” said, “a majority of current users … expressed an uncertainty over which questions Sentinel was best suited to answer. “

“In addition,” the report said, “a broad set of potential users continues to question the inherent value of observational data and remains unconvinced about how Sentinel outputs” can be used.

Rep. Rosa DeLauro (D-Conn.), who participated in the hearings on the FDA’s handling of drug safety a decade ago, told STAT she was deeply concerned that the agency was relying too heavily on Sentinel.

“While the Sentinel program was designed to fill a gap in FDA’s ability to track unsafe drugs, the initiative has spiraled into an unreliable mechanism for the agency to maintain the appearance of doing its job,” DeLauro said.

Moore, the scientist at the Institute for Safe Medication Practices, said he believes Sentinel can be useful to detect some side effects. But in other cases, he said, it could fail entirely because of its dependence on insurance billing codes.

“A heart attack would be accurately diagnosed and billed as such,” he said.

“But things like psychiatric side effects, digestive problems, cognitive impairment, and even acute liver failure — those sorts of things would be invisible.”