Which is a better proxy for estimating the Covid-19 infection rate in the United States: the infection rate in the NBA or the infection rate in China or Italy? The answer to that question reveals some of the pitfalls of diagnostic Covid-19 testing.
Kevin Durant, Rudy Gobert, and at least eight other NBA players have tested positive for Covid-19. The Jazz, Pistons, Nets, Lakers, and Celtics have all reported at least one positive test result for an active player, with 10 detected cases among the roughly 75 standard contract players on these teams. If this is representative of the rest of the NBA, then 13% of players are positive. Even if every other NBA team tested all its players and found no other positive results (a strong and doubtful assumption), the prevalence of Covid-19 in the league would be about 2%. And that is the low end of the possible range.
If 2% sounds low to you, here’s a reference point: Worldwide, at the time we write this, there have been 466,955 detected cases among 7.8 billion people. This translates into a cumulative incidence of 0.0060%. Because of limited testing, the actual number of cases is likely much higher. China and Italy have been hit the hardest, with approximately 82,000 cases identified in China (a crude incidence of 0.0059%) and 74,000 in Italy (a crude incidence of 0.12%).
Could the NBA have several thousand times the rate of Covid-19 compared to the rest of the world? Probably not. Either way, despite questions raised about the Covid-19 fatality rate, the epidemic is devastating major health care systems, including Italy and China. All eyes now are on New York City battling a surge in cases requiring ICU admissions.
This stunning discrepancy reveals a lot about the global approach to Covid-19 testing.
Our program has been tracking infections for years. In reviewing statements and reports of some policymakers, health experts, and the media, we worry about a collective amnesia for all we have learned about the difficulty of understanding the results of diagnostic testing. There are two broad issues worth understanding for Covid-19 diagnostic testing.
The first issue relates to who gets tested. Due to an extreme paucity of kits in the United States, testing has been done mainly on people with convincing symptoms or who have been in contact with people diagnosed with Covid-19. If testing is performed on a wider population sample, the incidence of Covid-19 will likely prove to be much higher than currently reported.
That would be both bad news and good news. The bad news is that it could mean that the pandemic is more advanced than we thought. The good news is that a higher number of Covid-19 infections would make the case fatality rate lower than current estimates because we would be dividing by a larger number of cases.
The second issue relates to the test itself. There is already a growing concern that the diagnostic test for Covid-19 is not perfect. It may come back positive in some people who are not infected with SARS-CoV-2 and negative in some people who are.
Diagnostic tests are developed with both sensitivity and specificity in mind. The greater the sensitivity, the less likely it will miss real cases. The greater the specificity, the more likely uninfected individuals will be correctly deemed negative.
The problem is that tests almost never have 100% sensitivity and 100% specificity. The test and the truth together create four possibilities: true positives, true negatives, false positives and false negatives. There’s a trade-off involved because an increasingly liberal test (more sensitive) will include more and more individuals in the population who do not actually have the disease (less specific). This trade-off has important implications for interpreting Covid-19 population trends based on testing to date and going forward.
For Covid-19 testing, the threshold for calling a test positive should not be set too high. Failing to isolate someone who actually has Covid-19 and sending her back to a nursing home would be put many people in harm’s way. But if the bar for calling a test positive is set too low, then a subset of patients who do not have Covid-19 will test positive for it.
Tests for Covid-19 are developed under idealized conditions with test tube samples from positive cases and negative controls. New ones are approved by the FDA under an emergency use authorization based on analytic validity, meaning they performed appropriately on test tube samples. Demonstration of clinical validity is not required.
Real-world performance could be worse. For example, the test could be cross-reactive with another virus. Or it could detect the presence of the novel coronavirus even after an individual is no longer infectious — so while it accurately detected the virus, it did not correctly identify the disease. Coronavirus is present in secretions in such abundance that it is easy to detect, potentially too easy: Even the most minuscule cross-contamination while samples are handled creates the risk of a false positive. Think of a hospital environment where personal protective equipment like masks are being reused due to shortages. Poor techniques in sample collection could also lead to false negatives.
Many diagnostic tests, even routine ones, are not rigorously validated against an external, real-world gold standard. The myriad new tests emerging for Covid-19 include at-home tests and rapid tests for point of service testing. They are produced by multiple vendors, each with different and as-yet-unmeasured accuracy. As serology testing for Covid-19 exposure and immunity is offered to the public, its false positive rates and false negative rates may be markedly different from the viral detection tests that have dominated to date.
When applied to a broad swath of the population, a test’s performance can be surprisingly counterintuitive. It can perform worse than expected, producing a potentially large proportion of false positives in populations less likely to have the disease. Consider a scenario with Covid-19 testing in an asymptomatic or mild population with 1 in 51 people infected (about 2%, the lower estimate for the NBA). Assume the test is always positive in individuals with the disease but falsely positive 10% of the time (which would be superior to many medical tests in use). As shown in the figure, the chance that someone with a positive test result is actually infected is under 20% (1 in 6).
As systematic testing is performed in the general population, patients less likely to have the disease — including asymptomatic individuals without known exposures — will be tested. A large number of false positive results would lead to an overestimate of the number of asymptomatic cases in many regions.
A large number of false positives could also overestimate the contribution of asymptomatic spread to the dynamics of the pandemic. False positives could also decimate the health care workforce if workers were inadvertently and unnecessarily quarantined and kept from seeing patients. The magnitude of the false positive and false negative problem remains unclear. Policymakers are aware of this potential issue, but early data on the sensitivity and specificity of tests in Wuhan, China, have been retracted.
There is also a counting problem. If the virus is widespread, which it may be, the more Covid-19 testing that is done the more cases of Covid-19 we will find. When the World Health Organization states that “it took 67 days from the first reported case to reach the first 100,000 cases of Covid-19, it took only 11 days for the second 100,000 cases, and just four days for the third 100,000 cases,” we must recognize that part of that rise may be due to increased testing.
There will soon be a drastically increased number of tests and testing platforms in the United States. Germany and South Korea have implemented drive-through testing as an approach. In Iceland, deCODE genetics recently released the results of a population sample of 5,571 Covid-19 tests, of which 48 were positive, enabling what is perhaps a more robust population estimate of 0.86% prevalence. An important caveat, though, is potential false positives and false negatives.
Back to the NBA. The discrepancy between the incidence of Covid-19 among professional basketball players versus the incidence in China and Italy can be explained in at least two ways: false positive test results in the NBA or basketball players’ privileged access to early Covid-19 testing, much of which was performed by private companies. It is not hard to imagine some false positive test results among NBA players. There has been considerable ire directed at the NBA for moving asymptomatic players to the head of the line while critically ill suspected cases go untested in the U.S. But if many more Americans than we have counted have mild or asymptomatic cases of Covid-19 and simply cannot get tested, perhaps the NBA experience might actually be teaching us about the true prevalence of Covid-19 in the general population.
Now that testing is coming on line at scale, a critical next step is to design a population-based sampling approach that includes accurate, individual-level, de-identified information about whether tested patients had clinical courses consistent with Covid-19, their comorbidities, medications, and, of course, survival. The clinical validity of the testing strategy is paramount. Before this information emerges, strategies to reduce spread including social distancing, ceasing nonessential activity, and closing schools remain essential. But we should rapidly assess the sensitivity and specificity of Covid-19 diagnostics and serology testing as well as their performance across different populations.
Let’s hope what we learn from increased testing in the general population is encouraging and leads to nimble and targeted policy making in the next phases of the pandemic.
Arjun K. Manrai, Ph.D. is in the Computational Health Informatics Program at Boston Children’s Hospital and is an assistant professor of pediatrics and biomedical informatics at Harvard Medical School. Kenneth D. Mandl, M.D. is director of the Computational Health Informatics Program at Boston Children’s Hospital and professor of pediatrics and biomedical informatics at Harvard Medical School.