Skip to Main Content

When Covid-19 began spreading in the United States, Emory Healthcare was prepared in one way other health systems were not: A decade earlier — in response to the 2009 H1N1 flu outbreak — it had built an online symptom checker that could be quickly adjusted to screen patients for the new respiratory illness.

The tool was launched on March 20, and in less than three weeks, more than 300,000 screenings had been completed, including many thousands in early hotspots such as California (18,500) and New York (17,000), according to data Emory provided to STAT. In nearly 22 percent of cases, patients reported signs of severe illness and were directed to seek emergency care.

While the rapid uptake is a sign of success, it also raises questions about how the symptom checker, and others like it, are affecting care and clinical resources during the pandemic. Is it helpful to have an automated tool in Georgia telling patients in New York to use emergency services that may already be overwhelmed? Are patients getting the right advice based on their symptoms and the circumstances in their communities?


The urgency of the crisis does not allow time for scientific studies to supply answers, at least not right away. But the data from Emory offer a first glimpse at the geographic reach of these tools, the symptoms most commonly reported, and the advice patients are getting from artificial intelligence that is being used for the first time in the throes of a public health crisis.

In New York City, Nicholas Genes, an emergency medicine physician at Mt. Sinai Hospital, has been following the evolution of the Emory tool and the advice it is giving to patients from hundreds of miles away. At first, he said, it seemed quick to refer people to emergency care but has become more nuanced in recent days. “It’s saying seek medical care or medical attention, and it’s giving you an array of options, including the emergency department.” he said. “They’ve done a lot of work to make it easy to use and incorporated a lot of the latest evidence and guidelines.”


To peel back the curtain on that daily work is to glimpse the challenges associated with calibrating digital tools during a crisis in which the use of technology is both a practical imperative and a potential hindrance, if implemented in the wrong way. The tool, dubbed c19check, requires constant updating by the Emory team to keep it current with rapidly changing clinical knowledge and treatment guidelines that sometimes change by the hour. That these caregivers are doing this work while simultaneously treating Covid-19 patients reflects their faith in the technology, but also a sense of urgency that comes from staring through a thin veil of plastic into the eyes of patients dying of a mysterious communicable disease. It could easily be them, or their family members.

That fear has caused Justin Schrager, a 38-year-old emergency medicine physician with a four-week-old baby at home, to sequester himself in his garage during the pandemic. He works an overnight shift in the Atlanta hospital system’s emergency department and returns home to sleep and work on the symptom checker in the afternoons. For him, c19check is a way to help as many patients as possible. It is also a way out of his garage.

“Pretty much everyone I know is sending their kids to live with cousins or staying in hotels,” Schrager said. “It’s just terrible from a personal standpoint what health care workers are doing. We see how bad this disease is. We see it in person and it’s scary as hell. We don’t want to get that disease and we sure don’t want to bring it home.”

Dr. Justin Schrager
Justin Schrager, an emergency medicine physician at Emory University Hospital, is part of the team working on the c19check symptom checker. Emory University Hospital

In addition to the steep learning curve with the new disease, one of the biggest challenges hospitals face in any public health crisis is the sudden flood of visitors to their emergency departments. Patients with real and imagined symptoms arrive in overwhelming waves, leaving health care workers unable to tend to all the newly sick on top of the normal volume of heart attacks, broken bones, and car accidents. From that standpoint, Schrager said, the best thing he and his colleagues could do was to use the symptom checker to dispense as much advice as possible from afar, and help people detect worrisome trends in their symptoms.

“Coronavirus is an incredible disease,” he said. “You’re a little bit sick for a week or longer and then you’re just catastrophically ill. It’s not predictable. If you can take action early, and get them into the right care setting or at least monitor them, you can prevent them from decompensating past the point of where we can really help them.”

So far, Emory’s tool, which asks people to type in symptoms and underlying health conditions, is advising the vast majority of users that they can ride it out at home, according to the health system’s data through April 7. About 40 percent are being told their symptoms indicate no risk of Covid-19, while about 38 percent are bucketed into intermediate and low risk categories and told to monitor their symptoms; another 22 percent are being directed to seek immediate medical attention.

The percentage falling into that latter category has increased since the first 10 days of use, when about 17 percent were advised to seek emergency services.

The most commonly reported symptoms by the more than 300,000 screened are cough (95,533); sore throat (88,756); headache (84,502); and fatigue (76,670), with a fairly even split among body aches, breathlessness, fever, and diarrhea.

It is difficult to determine how accurate the symptom checker is in assessing risk and triaging patients, because the users are spread all over the United States and the world. The largest number of completed screenings is in Georgia, at about 50,000, but many thousands have also been logged in Texas, Florida, and Illinois, in addition to the large numbers in California and New York. The site has been translated into 18 languages and is being used in countries from Estonia to Pakistan.

Emory’s physicians and researchers are devising a follow-up study to validate the tool by tracking users’ symptoms and health care utilization over time. They are also exploring ways to analyze the data to help inform treatment and public health approaches in the months ahead.

Their symptom checker is among dozens of similar tools launched in recent weeks by hospitals and technology companies across the United States. Large health systems such as Cleveland Clinic, Washington-based Providence St. Joseph Health, and the University of Southern California have created tools, as have insurers and digital health companies such as Buoy Health, Memora Health, and the telehealth company Ro. The CDC has launched multiple symptom checkers, one built with Microsoft and another with Apple.

symptom checker map
The top map is a heat map showing the number of users of Emory’s symptom checker around the U.S. The bottom is a heat map of users with fever, cough, and shortness of breath, a common set of Covid-19 symptoms. Emory University Hospital

Genes, the emergency medicine physician from Mount Sinai, said the fragmentation is not ideal, as it may create confusion and undermine the quality of care. But he also said that, during the pandemic, it is important not to make the perfect the enemy of the good.

“At some point, I do hope there is standardization, or at least harmonization,” he said. “But for right now I’m satisfied with the availability of the tools. We don’t want to expose people unnecessarily to the emergency department or consume PPE.”

The idea for an online symptom checker was born at Emory more than a decade ago, when H1N1 began causing chaos in emergency departments around the country. Emergency medicine physicians developed a system called SORT (Strategy for Off-Site Rapid Triage). Initially it was meant to create a a set of algorithms that would allow minimally trained health care workers to screen people for signs of H1N1, but it evolved into a web-based tool to help people screen themselves.

In October 2009, as H1NI was tailing off and the seasonal flu season ramping up, the Centers for Disease Control and Prevention posted a slightly modified version of the tool on its website, as did Microsoft and the federal Department of Health and Human Services.  During that flu season, the sites collectively logged more than 2.5 million visits. But other than counting web hits, the makers of the tool were unable to gauge its effect on patient safety or the level of care sought by users.

A study published on its initial use in 2010 said its effects on patient care needed further study, but it predicted, almost eerily, that it could be quickly adjusted to respond to future epidemics involving flu-like illnesses such as an earlier coronavirus: SARS. “SORT-like algorithms for selected public health threats such as severe acute respiratory syndrome could be even be prepared and evaluated in advance and deployed if needed,” the authors wrote. “This method could help reassure a nervous public, particularly in the early phases of an outbreak when many people otherwise rush to the nearest ED.”

That moment arrived in Atlanta in early March, as the coronavirus began  preading in the United States.

Support STAT: If you value our coronavirus coverage, please consider making a one-time contribution to support our journalism.

At Emory, a team that included Schrager and Anna Quay Yaffee, director of global health in emergency medicine at Emory, set to work adjusting the H1N1 symptom checker to Covid-19. They reasoned online screenings should be fast — less than 30 seconds — and focus on advising people what level of care they should seek.

Dr. Anna Yaffee Headshot
Anna Quay Yaffee, director of Global Health in Emergency Medicine at Emory, also works on the symptom checker. Emory University Hospital

The biggest challenge was how to calibrate the algorithm to give people the right advice. “The app is designed to be conservative without being overly conservative,” said Schrager, who acknowledged that it may result in people visiting ERs that have a higher threshold on how sick patients should be before visiting the hospital.

One physician in New York contacted the team to let them know that ERs in the city did not necessarily have the resources available to care for every patient. Emory has updated the tool to include more options for getting advice and care, such as hotlines established by state and local health departments.

“At the end of the day,” Schrager said, “those people all still need an evaluation by a doctor, possibly an ER doctor, and that shouldn’t necessarily be contingent on the local resources.”

The Emory team, which has based the tool on CDC guidelines, also needed to decide which questions to ask and in what order to accurately assess a person’s risk level. They quickly determined the rapid community spread of the illness made travel questions irrelevant, but it was more challenging to decide which symptoms to ask about, and how.

“It doesn’t make sense to ask people if they have a cough because some people have a cough all the time,” Schrager said. “The real question you want to ask is if it is a new cough, and what kind of cough.”

They spent a considerable amount of time debating whether to include a question about red itchy eyes, which is among a long list of coronavirus symptoms documented in clinical studies.

“It was very controversial,” Yaffee said, laughing at the memory of the conversation. “I mean we spent hours on it — hours.”

They ultimately decided to leave it off, and have updated the system on a regular basis, to both add and remove symptoms based on shifts in clinical knowledge, and change how questions are asked if they seem to be having a disproportionate impact on the tool’s recommendations.

While dealing with those changes, Schrager and Yaffee are also going through a similar process of reinvention every time they report to the ER. They’ve had to rethink almost everything about their daily routines.

Yaffee said she brings in extra scrubs to change into so she doesn’t bring contaminated clothing outside the hospital. She doesn’t bring food in because she doesn’t want to handle or consume it there. During her shifts, she has to think carefully about how often to wash her hands, what to touch, and when it is safe to take off her mask. She uses a UV light to disinfect her phone, key and ID badge, and takes a long shower when she gets home.

“It’s a lot of mental exertion and it’s pretty exhausting,” she said. “And then we see these really, really sick patients who shouldn’t be sick, that don’t have other reasons to be sick. And their families can’t see them. People are dying without their families there. It’s really hard to watch that.”

But in the darkness of the pandemic, they are warmed by strangers who cheer them and drop off food. They have formed an uncommon esprit de corps with their colleagues, a hopefulness amid shared struggle that binds them together in the ER, during technology troubleshooting sessions, and in the moments in between, when coronavirus presses on their minds like a crushing weight.

“Everybody is there to help and is doing the best they can,” Yaffee said. “It’s the only thing that gets me through these shifts.”

  • I reviewed most of these tools in regards to COVID19, however, none of them have good statistical model to go with it. I have a web app based on my statistical model and data from CDC and WHO. It also balances when to call 911 and when to call your doctor based on current recommendations. URL is: Works on mobile devices as well. Any feedback will be highly appreciated. It is work in progress. Volunteers are welcomed.

Comments are closed.