The world is in a rush to find ways to fight Covid-19. This urgency makes sense for a pandemic that has killed more than a half-million people since it began in late December. But if the quality of these solutions is subpar, then users may turn away not only from these faulty solutions but may lose confidence in broader efforts and scientific development to combat Covid-19, putting public health at significant risk.
Take the more than 100 coronavirus tests the Food and Drug Administration has rushed to approve as a potential solution to the pandemic. Some fail to identify up to 20% of coronavirus cases. Antibody tests, designed to detect whether an individual has developed antibodies in response to a coronavirus infection, are even less accurate: Up to half of people who test positive don’t actually have the antibodies.
I am particularly interested in the quality of the apps that technology companies have been pushing as their solutions to the pandemic. A growing number of contact tracing apps are being designed to track and reduce the spread of coronavirus. These apps leverage Bluetooth proximity information or location data to detect and notify an app user if they were exposed to another app user who was later diagnosed with Covid-19.
Yet concerns about how effective these apps will be have led public health agencies in the U.S. to hesitate to adopt them. Even in countries that are using coronavirus apps extensively in their response to the pandemic, it can be difficult to isolate their precise impact from other measures used to prevent the virus from spreading.
Questions about quality and accuracy may compound Americans’ existing wariness toward tracking technologies like contact tracing apps. Yet for theses apps to work, they need to be adopted by most of the population: Their benefit increases exponentially with the number of users.
This presents a circular problem: The effectiveness of these apps will inevitably influence whether people are willing to install them, while the number of people who install the app will directly influence its effectiveness.
How effective is effective enough for Americans to want to install these coronavirus contact tracing apps? In early May, my colleagues and I at Microsoft Research crowdsourced a survey to 3,826 Americans to answer that question.
Using the results of the survey, we were able to gauge the relationship between effectiveness and installing the app: For every 1% it reduced the infection rate, Americans would be 5% more likely to install the app.
How effective do these apps need to be to reach majority adoption? Our results suggest that more than 60% of Americans would install an app that reduces their infection rate by 50%, from 30 per 1,000 people to 15 per 1,000, and more than three-quarters of Americans would be willing to install an app that reduces their infection rate by 97%, to 1 in 1000.
Reducing the infection rate is a key component of what it means for coronavirus apps to work and to speed our ability to return to normal. Of course, the effectiveness of these apps depends on more than their accuracy. It also depends on whether users self-isolate and get tested when the app requests they do so.
Like coronavirus tests, contact tracing apps are not perfectly accurate. There is always a chance of false negatives — the app failing to detect when you were exposed to someone infected with coronavirus — or false positives — the app notifying you that you have been exposed when you haven’t been. These errors are not only irritating, but research in other domains finds that low accuracy can cause users to ignore app warnings, reducing the app’s effect. In addition, accuracy problems cause people to abandon coronavirus apps or not install them in the first place.
But unlike coronavirus tests, no agency evaluates and approves these apps. There is no requirement for app developers to prove their apps are effective or report their accuracy. Setting the accuracy bar too low may lead to users fleeing the product, making it difficult to get new adopters as word spreads. The Care19 coronavirus tracing app released in North Dakota has already suffered from this problem.
How high should the bar be set for the accuracy of coronavirus apps? Our survey results suggest that an app must be able to accurately detect at least 50% of coronavirus exposures and must raise false alarms less than 10% of the time before most Americans will adopt it.
A crisis like this raises an important question: How good is good enough in the time of coronavirus? While regulators and technology companies may feel that the pandemic requires a rush to the finish line, just developing medical treatments and tracking apps isn’t enough. Initial data suggest that Americans’ are not on board for using inaccurate or ineffective resources. That means medical researchers and app developers have their work cut out for them. Not only must the tests and treatments they are making be effective and safe — and the apps both easy to use and protect people’s private data — Americans are setting a high bar for how well these solutions must work before they are willing to adopt them.
Elissa M. Redmiles is a security and privacy researcher at Microsoft Research and the Max Planck Institute for Software Systems.