Skip to Main Content
Contribute Try STAT+ Today

A widely followed model for projecting Covid-19 deaths in the U.S. is producing results that have been bouncing up and down like an unpredictable fever, and now epidemiologists are criticizing it as flawed and misleading for both the public and policy makers. In particular, they warn against relying on it as the basis for government decision-making, including on “re-opening America.”

“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington.

Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”

advertisement

The IHME projections were used by the Trump administration in developing national guidelines to mitigate the outbreak. Now, they are reportedly influencing White House thinking on how and when to “re-open” the country, as President Trump announced a blueprint for on Thursday.

The chief reason the IHME projections worry some experts, Etzioni said, is that “the fact that they overshot will be used to suggest that the government response prevented an even greater catastrophe, when in fact the predictions were shaky in the first place.” IHME initially projected 38,000 to 162,000 U.S. deaths. The White House combined those estimates with others to warn of 100,000 to 240,000 potential deaths.

advertisement

That could produce misplaced confidence in the effectiveness of the social distancing policies, which in turn could produce complacency about what might be needed to keep the epidemic from blowing up again.

Believing, for instance, that measures well short of what China imposed in and around Wuhan prevented a four-fold higher death toll could be disastrous.

The most notable bounces in the IHME projections have been for the eventual total of U.S. deaths by early August, which is when many epidemiologists believe the outbreak will be tailing off. (Many expect daily deaths in the U.S. to fall to 10 or fewer by early June, from 2,000 or so in April.) Death projections for individual states have also fluctuated significantly.

The IHME website explains that, “As data continue to come in, our estimates may change. Specifically, new death data … have changed our projections.”

Its model differs from those used by almost all other epidemiologists.

There are two tried-and-true ways to model an epidemic. The most established, dating back a century, calculates how many people are susceptible to a virus (in the case of the new coronavirus, everyone), how many become exposed, how many of those become infected, and how many recover and therefore have immunity (at least for a while). Such “SEIR” models then use what researchers know about a virus’s behavior, such as how easily it spreads and how long it takes for symptoms of infection to appear, to calculate how long it takes for people to move from susceptible to infected to recovered (or dead).

“The fundamental concept of infectious disease epidemiology is that infections spread when there are two things: infected people and susceptible people,” Lipsitch said.

Newer, “agent-based models” are like the video game SimCity, but with a rampaging pathogen: using computing power unimagined even a decade ago, they simulate the interactions of millions of individuals as they work, play, travel, and otherwise go about their lives. Both of these approaches have often nailed projections of, for instance, U.S. cases of seasonal flu.

Support STAT: If you value our coronavirus coverage, please consider making a one-time contribution to support our journalism.

IHME uses neither a SEIR nor an agent-based approach. It doesn’t even try to model the transmission of disease, or the incubation period, or other features of Covid-19, as SEIR and agent-based models at Imperial College London and others do. It doesn’t try to account for how many infected people interact with how many others, how many additional cases each earlier case causes, or other facts of disease transmission that have been the foundation of epidemiology models for decades.

Instead, IHME starts with data from cities where Covid-19 struck before it hit the U.S., first Wuhan and now 19 cities in Italy and Spain. It then produces a graph showing the number of deaths rising and falling as the epidemic exploded and then dissipated in those cities, resulting in a bell curve. Then (to oversimplify somewhat) it finds where U.S. data fits on that curve. The death curves in cities outside the U.S. are assumed to describe the U.S., too, with no attempt to judge whether countermeasures —lockdowns and other social-distancing strategies — in the U.S. are and will be as effective as elsewhere, especially Wuhan.

“We are becoming more confident that the shape of the curve [is accurately] informed by locations outside the U.S.,” said Theo Vos, professor of health metrics science at IHME.

According to a critique by researchers at the London School of Hygiene & Tropical Medicine and Imperial College London, published this week in Annals of Internal Medicine, the IHME projections are based “on a statistical model with no epidemiologic basis.”

“Statistical model” refers to putting U.S. data onto the graph of other countries’ Covid-19 deaths over time under the assumption that the U.S. epidemic will mimic that in those countries. But countries’ countermeasures differ significantly. As the epidemic curve in the U.S. changes due to countermeasures that were weaker or later than, say, China’s, the IHME modelers adjust the curve to match the new reality.

Each run of the model, updated with new U.S. data, produces estimates of future and total deaths, ICU use, and other outcomes, with uncertainty bounds. That is, IHME says the actual number of deaths and other outcomes has a 95% likelihood of falling between a stated upper limit and lower limit. In late March, for example, IHME projected that there will be a total of 81,114 Covid-19 deaths in the U.S. over the next four months, but that number came with a caveat: The actual number could be as few as 38,242 and as many as 162,106.

“This appearance of certainty is seductive when the world is desperate to know what lies ahead,” Britta Jewell of Imperial College and her colleagues wrote in their Annals paper. But the IHME model “rests on the likely incorrect assumption that effects of social distancing policies are the same everywhere.” Because U.S. policies are looser than those elsewhere, largely due to inconsistency between states, U.S. deaths could remain at higher levels longer than they did in China, in particular.

While other epidemiologists disagree on whether IHME’s deaths projections are too high or too low, there is consensus that their volatility has confused policy makers and the public:

— Last week IHME projected that Covid-19 deaths in the U.S. would total about 60,000 by August 4; this week that was revised to 68,000, with 95% certainty that the actual toll would be between 30,188 and 175,965.

— On March 27, it projected that New York would see 10,243 deaths (and that the total had a 95% chance of falling between 5,167 to 26,444) by early August. Three days later, the New York projection was 15,546, and on April 3 it was 16,262, Jewell and her colleagues pointed out in another analysis, published in JAMA on Thursday.

— On April 8, IHME projected 5,625 deaths for Massachusetts by August; on April 13, it was 8,219.

Such changes, Vos said, “are well within the uncertainty bounds we predicted.” In addition to reflecting more recent data, the projections are now based on a moving average of daily deaths rather than one-day numbers.

Although IHME says its approach has always been to change projections when new data become available, critics say that underlines the model’s flaws, namely its need to constantly re-calibrate rather than, as true epidemiology models do, use basic outbreak parameters such as a disease’s infectiousness to project the course of an epidemic in a way that policy makers can use as a lode star, not a strobe light that flares and dims repeatedly.

“Since they started with very little U.S. data, when they add some, their projections move a lot,” said the Hutch’s Etzioni.

Even the predictions of daily deaths “have been highly inaccurate,” said statistician Sally Cripps of the University of Sydney, who led a team that examined IHME’s up-and-down projections. “It performs poorly even when it predicts the number of next-day deaths: The true number of next-day deaths has been outside the 95% intervals 70% of the time.” If the 95% calculation correctly reflects a model’s uncertainty, then textbook statistics say the true numbers can fall outside that range no more than 5% of the time.

Lipsitch and some other experts worry that by failing to include disease transmission, IHME’s projections of deaths could be too low. But more and more models are projecting a less dire future. Three weeks ago a SEIR model from researchers at the Massachusetts Institute of Technology projected that total U.S. cases will plateau later this week, reaching 600,000 and then adding ever-fewer cases each day. So far it’s pretty much on the money, with the U.S. case count at 650,000 on Thursday and new daily cases remaining mostly flat.

A different, data-driven model from researchers at the University of Washington predicts “about 1 million cases in the U.S. by the end of the epidemic, around the first week in June, with new cases peaking in mid-April,” said UW applied mathematician Ka-Kit Tung, who led the work. “By the first week of June, we project that the number of new cases will be close to zero if current social distancing policies are maintained.” That model predicted two weeks ago that the number of new daily cases would peak around now, as seems to be the case.

Helen Branswell contributed reporting.

This story has been updated to include the earliest IHME projections.

  • “It performs poorly even when it predicts the number of next-day deaths: The true number of next-day deaths has been outside the 95% intervals 70% of the time.” If the 95% calculation correctly reflects a model’s uncertainty, then textbook statistics say the true numbers can fall outside that range no more than 5% of the time.

    I read the description on the University of Washington Institutional Review Board website of projects that require IRB approval and I couldn’t figure out these statistical modelers could have avoided submitting their project to ethical review. Based on the quote, they don’t even seem to have a valid statistical model. Their approach clearly is experimental and still, they are offering the results directly to public officials who will determine human policies, who gets what, and more. I ordinarily wouldn’t think anything of it, but it was really very surprising to see these researcher using their experimental models to advocate for the arrest of anyone venturing from their homes. The researchers seemed to have lost sight of their proper ethical role.

  • This is the first time such a “Hot Zone” Incident has happened in USA and everyone was unprepared, and this is just a Mild to Moderate severity as opposed to an Ebola outbreak (scale of event)! That being said, this wake up call tells us we have to weigh opposite sides of this Event while adding a human element into this new equation to better prepare for the next cycle of this pandemic. Here are some disconcerting points of contention and how can we add this non human element into an equation to balance erroneous results?
    1. POTUS came out with a talking point that this Coronavirus was fake news created by the other political party? This created (and still does) an attitude that it should not be taken seriously by even the most well read citizens. Quantify this element into an equation to come up with reasonable numbers for the model. Impossible! All of this time is lag time, meaning the time taken to ignore and correct in mid stream the data coming out to solifify an actual problem.
    2. What was the time between WHO receipt of information to USA actually enforcing some type of controls (whether quarentine or curfew, or both on an individual basis). My time zone was about 1 month ( around March 15? EST Midwest) so what happened to USA between January 15- March 14?
    3. Currently, there are some states who refuse to initially self quarentine and still refusing to do so. If they do, a few days ago, people in Ohio, WI, PA, etc refusing to follow these quarantine guideline/curfew and this jeopardizes other peoples’ health. How do we quantify this as part of equation to come up with a central accurate number. I am sure that if any action was not followed as of 6-8 weeks ago, we would have had 2-3x the deaths of the low figure we have today
    4. How do we account for POTUS telling individual states that their governors are in control while praising those who go against the states’ quarentine/curfew guidelines. These are mixed messages and do more to cause suspicion and confusion for the citizensand this raises the risk profile for all in those surroundings.
    5. There is no doubt that economic considerations are cause for concern but that also tells us the monetarization of services (health care, supply chain, retail and others) have created a ‘black hole’ where all will suffer equally due to the lack of safety nets in the present societal mobilization milieu.
    6. How do we model much of what I see as human variables and add them to numerical data to come up with an accurate model. Keep in mind that if even the curfew/self quarentine is lifted, the following is noted that we will still be 2-3 week behind the curve, add the concompliance of x states who refuse to follow the guidelines, the lack of current tests to indicate ‘positiev” or ‘negative” status and the rush to get the economy on track.

  • When you started quoting the Imperial College as a critic of the model…I couldn’t stop laughing. Frankly all of these models have become god speak. But no one is look at the economic fall out. Maybe its time to start finding a check and balance on these epidemiologists trying to make policy for the US. My favorite is we’ve kept big box stores open so people can get food – but stopped gatherings of 50 or more. I gotta say having gone to my local wal mart today – if that isn’t a gathering of 50 or more – I don’t know what is.

  • The travesty is that science is being used to make public policy. Science is important, but it is only one aspect. Issues like overall public health, the economy, it’s effect on mental health, human rights, and morality/ethics are being ignored because the public has been terrified by a virus that probably kills far less than 1% who are infected and most off those were un-healthy and wouldn’t have lived much longer anyway. It’s time to take a step back and try to undo all the damage that has already been done by shutting down so much of public life and the economy.

    • Steve – your claim it “probably” kills “far less than 1%” would be great news, to me – but it would be news to me nonetheless – I can find nothing to back that up. Wishful thinking which is not panning out so far. Do you have any actual evidence for that?

    • This coronavirus is serious! What would you think if quarentine/curfew guidelines were not followed? We would definitely have 2-3x the current deaths that are now being reported.
      Data are needed as part of the epidemiological survey as a guide to ascertain equipment, beds, medication (if any) etc to see a direction so we can notify the greater public of what needs to be done to mitigate this ‘flu virus’. The mere fact of the setting up of additional facilities per data collection tells us something is going on and we need to get a handle on it ASAP.
      Those who do not learn from events (history) will repeat them and no one will be the wiser. People talk about making US America great again but this falls on deaf ears. For the first time in USA. I have seen people (3 weeks ago!) lining up at Wallmart, Sams Club, etc waiting in line to get toilet paper, food, etc anf I though this was Russia or some Communist or Socialist country. Most of the people were 6 feet apart, trying to get basic food stuff when many stores had empty shelves, had no idea when stock would be in the store.
      This is definitely not your basic flu epidemic.

  • I’m a CDC reporting physician and if remdisivir is widely used it would alter predictions greatly. And obviate the need for panic shutdowns. Clear out hospitals of virus victims. Save many lives. Same with hydroxychloquine and invermectin. But those epidemiologists would be out of jobs.

    • A good friend of mine always said when it comes to public policy…follow the money. There sure are a lot of need for clinical trials right now. Curious that.

  • The people who created the IHME study should be fire – it is laughable. They state that they have a 95% certainty that by august 4th the death toll will be between 30000 and 176000??? WTF

    I went to a horse race once where there were 10 horses racing – I predicted that between 0 and 10 horses would finish the race. That prediction is about as useful.

    Why don’t you take daily new case in the U.S. of 30,000 which has been constant over the last 20 days with next to no testing of the population and multiply it by 2% death rate (which is way lower than the 5% + death rate seen everywhere) – and you get 600 deaths per day lowball minimum for the next 2 years or so. This would put us at 104,000 by august 4th based on 1/2 the proven death rate.

  • I stopped reading here

    ” The most established, dating back a century, calculates how many people are susceptible to a virus (in the case of the new coronavirus, everyone)”

    Wha’s he basis for the everyone claim? Absolutely nothing.
    With 80% of positive testing people asymptomatic, you can’t even make a correlation between the clinical presentation of the disease and a positive test.

    There are people testing negative that still have symptoms and returning to the hospital.

    Positive cases versus deaths has a divergence that is INCREASING>

    Like seriously people. The data available is horrible the models are horrible. No one can make any predictions.

    We need to START OVER and re-diagnose the patient.

    • I’ve watched almost every WH briefing and every time I see Dr. Fauci he says that he relies on data, not models (generally referring to the IMHE model as it is the most often used as a basis for questions). And, Dr. Birx states they look at several models and models of models. So why would the author state that the IMHE model is reportedly being used as a tool to reopen the country? This whole article is based on a false premise. Useless.

  • Vince: Are frontline first responders and health care workers putting themselves at risk — and dying — to save lives not your concern? Health care systems are already overwhelmed by this virus, which means that cancer and heart disease and accident patients may not get the care they need. If we don’t do what we can to protect frontline workers, how can we expect them to be there for us?

  • This model has to be adjusted by density…. setting a 1 per million IFR rate for everyone appears to be a great metric in aggregate but ignores the law of large numbers. North Dakota doesn’t have a population of 1 million people. There are 11 people per mile in the state. While New York City has over 62,000 people per mile. Yet the model has New York coming out of social distancing in late May but North Dakota after June 8th. The model appears to overestimate the need for distancing in low density states, but may also be ignoring the impact of high density population sets.

    • @Michael Bartholomew : It helps to know what model are you referring to.

      IHME predicted peaks for North Dakota April 19 @ 11:00 EDT :
      Hospital resource use: May 4
      Peak deaths: May 5

      COVID-19 started early in New York than in North Dakota, and a lower population density should result in slower transmission.

Comments are closed.