Which is a better proxy for estimating the Covid-19 infection rate in the United States: the infection rate in the NBA or the infection rate in China or Italy? The answer to that question reveals some of the pitfalls of diagnostic Covid-19 testing.
Kevin Durant, Rudy Gobert, and at least eight other NBA players have tested positive for Covid-19. The Jazz, Pistons, Nets, Lakers, and Celtics have all reported at least one positive test result for an active player, with 10 detected cases among the roughly 75 standard contract players on these teams. If this is representative of the rest of the NBA, then 13% of players are positive. Even if every other NBA team tested all its players and found no other positive results (a strong and doubtful assumption), the prevalence of Covid-19 in the league would be about 2%. And that is the low end of the possible range.
If 2% sounds low to you, here’s a reference point: Worldwide, at the time we write this, there have been 466,955 detected cases among 7.8 billion people. This translates into a cumulative incidence of 0.0060%. Because of limited testing, the actual number of cases is likely much higher. China and Italy have been hit the hardest, with approximately 82,000 cases identified in China (a crude incidence of 0.0059%) and 74,000 in Italy (a crude incidence of 0.12%).
Could the NBA have several thousand times the rate of Covid-19 compared to the rest of the world? Probably not. Either way, despite questions raised about the Covid-19 fatality rate, the epidemic is devastating major health care systems, including Italy and China. All eyes now are on New York City battling a surge in cases requiring ICU admissions.
This stunning discrepancy reveals a lot about the global approach to Covid-19 testing.
Our program has been tracking infections for years. In reviewing statements and reports of some policymakers, health experts, and the media, we worry about a collective amnesia for all we have learned about the difficulty of understanding the results of diagnostic testing. There are two broad issues worth understanding for Covid-19 diagnostic testing.
The first issue relates to who gets tested. Due to an extreme paucity of kits in the United States, testing has been done mainly on people with convincing symptoms or who have been in contact with people diagnosed with Covid-19. If testing is performed on a wider population sample, the incidence of Covid-19 will likely prove to be much higher than currently reported.
That would be both bad news and good news. The bad news is that it could mean that the pandemic is more advanced than we thought. The good news is that a higher number of Covid-19 infections would make the case fatality rate lower than current estimates because we would be dividing by a larger number of cases.
The second issue relates to the test itself. There is already a growing concern that the diagnostic test for Covid-19 is not perfect. It may come back positive in some people who are not infected with SARS-CoV-2 and negative in some people who are.
Diagnostic tests are developed with both sensitivity and specificity in mind. The greater the sensitivity, the less likely it will miss real cases. The greater the specificity, the more likely uninfected individuals will be correctly deemed negative.
The problem is that tests almost never have 100% sensitivity and 100% specificity. The test and the truth together create four possibilities: true positives, true negatives, false positives and false negatives. There’s a trade-off involved because an increasingly liberal test (more sensitive) will include more and more individuals in the population who do not actually have the disease (less specific). This trade-off has important implications for interpreting Covid-19 population trends based on testing to date and going forward.
For Covid-19 testing, the threshold for calling a test positive should not be set too high. Failing to isolate someone who actually has Covid-19 and sending her back to a nursing home would be put many people in harm’s way. But if the bar for calling a test positive is set too low, then a subset of patients who do not have Covid-19 will test positive for it.
Tests for Covid-19 are developed under idealized conditions with test tube samples from positive cases and negative controls. New ones are approved by the FDA under an emergency use authorization based on analytic validity, meaning they performed appropriately on test tube samples. Demonstration of clinical validity is not required.
Real-world performance could be worse. For example, the test could be cross-reactive with another virus. Or it could detect the presence of the novel coronavirus even after an individual is no longer infectious — so while it accurately detected the virus, it did not correctly identify the disease. Coronavirus is present in secretions in such abundance that it is easy to detect, potentially too easy: Even the most minuscule cross-contamination while samples are handled creates the risk of a false positive. Think of a hospital environment where personal protective equipment like masks are being reused due to shortages. Poor techniques in sample collection could also lead to false negatives.
Many diagnostic tests, even routine ones, are not rigorously validated against an external, real-world gold standard. The myriad new tests emerging for Covid-19 include at-home tests and rapid tests for point of service testing. They are produced by multiple vendors, each with different and as-yet-unmeasured accuracy. As serology testing for Covid-19 exposure and immunity is offered to the public, its false positive rates and false negative rates may be markedly different from the viral detection tests that have dominated to date.
When applied to a broad swath of the population, a test’s performance can be surprisingly counterintuitive. It can perform worse than expected, producing a potentially large proportion of false positives in populations less likely to have the disease. Consider a scenario with Covid-19 testing in an asymptomatic or mild population with 1 in 51 people infected (about 2%, the lower estimate for the NBA). Assume the test is always positive in individuals with the disease but falsely positive 10% of the time (which would be superior to many medical tests in use). As shown in the figure, the chance that someone with a positive test result is actually infected is under 20% (1 in 6).
As systematic testing is performed in the general population, patients less likely to have the disease — including asymptomatic individuals without known exposures — will be tested. A large number of false positive results would lead to an overestimate of the number of asymptomatic cases in many regions.
A large number of false positives could also overestimate the contribution of asymptomatic spread to the dynamics of the pandemic. False positives could also decimate the health care workforce if workers were inadvertently and unnecessarily quarantined and kept from seeing patients. The magnitude of the false positive and false negative problem remains unclear. Policymakers are aware of this potential issue, but early data on the sensitivity and specificity of tests in Wuhan, China, have been retracted.
There is also a counting problem. If the virus is widespread, which it may be, the more Covid-19 testing that is done the more cases of Covid-19 we will find. When the World Health Organization states that “it took 67 days from the first reported case to reach the first 100,000 cases of Covid-19, it took only 11 days for the second 100,000 cases, and just four days for the third 100,000 cases,” we must recognize that part of that rise may be due to increased testing.
There will soon be a drastically increased number of tests and testing platforms in the United States. Germany and South Korea have implemented drive-through testing as an approach. In Iceland, deCODE genetics recently released the results of a population sample of 5,571 Covid-19 tests, of which 48 were positive, enabling what is perhaps a more robust population estimate of 0.86% prevalence. An important caveat, though, is potential false positives and false negatives.
Back to the NBA. The discrepancy between the incidence of Covid-19 among professional basketball players versus the incidence in China and Italy can be explained in at least two ways: false positive test results in the NBA or basketball players’ privileged access to early Covid-19 testing, much of which was performed by private companies. It is not hard to imagine some false positive test results among NBA players. There has been considerable ire directed at the NBA for moving asymptomatic players to the head of the line while critically ill suspected cases go untested in the U.S. But if many more Americans than we have counted have mild or asymptomatic cases of Covid-19 and simply cannot get tested, perhaps the NBA experience might actually be teaching us about the true prevalence of Covid-19 in the general population.
Now that testing is coming on line at scale, a critical next step is to design a population-based sampling approach that includes accurate, individual-level, de-identified information about whether tested patients had clinical courses consistent with Covid-19, their comorbidities, medications, and, of course, survival. The clinical validity of the testing strategy is paramount. Before this information emerges, strategies to reduce spread including social distancing, ceasing nonessential activity, and closing schools remain essential. But we should rapidly assess the sensitivity and specificity of Covid-19 diagnostics and serology testing as well as their performance across different populations.
Let’s hope what we learn from increased testing in the general population is encouraging and leads to nimble and targeted policy making in the next phases of the pandemic.
Arjun K. Manrai, Ph.D. is in the Computational Health Informatics Program at Boston Children’s Hospital and is an assistant professor of pediatrics and biomedical informatics at Harvard Medical School. Kenneth D. Mandl, M.D. is director of the Computational Health Informatics Program at Boston Children’s Hospital and professor of pediatrics and biomedical informatics at Harvard Medical School.
Great article…many thanks. You have nailed it. Can we believe a positive or a negative.
“Now that testing is coming on line at scale, a critical next step is to design a population-based sampling approach that includes accurate, individual-level, de-identified information about whether tested patients had clinical courses consistent with Covid-19, their comorbidities, medications, and, of course, survival.”
Following your advice is key. WHO or CDC or NIH….any group with credibility should develop the appropriately methodology for what the test sample should look like. The current criteria BIAS the information since we are mostly testing the SICKEST of the SICK.
Please keep us informed about developments.
THANK you…
I think the point is that testing is not really designed to help individuals. Testing is really more a statistical tool to analyze mass populations than a useful individual diagnostic tool – at least while we lack COVID-19 specific drugs and vaccines.
So testing is great for abstract epidemiological study of how a disease is progressing through a population. Testing is also very helpful in removing hysteria and politics from large scale response planning.
But if future COVID-19 specific drugs have possible significant adverse reactions…we will want two factor confirmation if possible instead of just relying solely on RNA reagent based tests. Specifically expert electron microscope confirmation as that “eyeballs on virus” identification is probably the most accurate follow-up confirmation of a positive test result.
That’s true however, not the total truth. Yes the tests are used to keep stats for analysis. But they are for the individual as well – and in a big way. For example, several states have developed new processes whereas treatment for COVID-19 will not cost if the person is uninsured or underinsured, etc. So if you are uninsured in my state, the test and the treatment won’t bankrupt you. At the same time, you will get treatment when needed for the virus…treatment that many uninsured simply cannot afford. People who are uninsured or underinsured will go for testing and treatment because they know in advance that the state is helping them with the costs. This also means that they may test positive and KNOW that they have the virus and thus better quarantine than if they didn’t get tested because they could not afford it. They could also get treatment which might be life-saving, whereas they may NOT get that treatment if they couldn’t afford it. So we are allowing those people to get tested and get treatment – possibly saving them and others – only because they had this help with payment. It’s a circle, I know. But the tests are necessary for these individuals. However, at this time there aren’t many people getting tested here because they are getting denied tests – even when symptomatic. Having said that, the numbers being reported are useless in my eyes. We can report “123 new cases” today but there are another 300 who are symptomatic but could not get tested. So who knows whether those 300 have the virus or not? So in my mind, why report that we have 123 cases when that number is virtually meaningless?
Too many people take testing negative as an excuse to break quarantine and return to normal life patterns. Testing does not prevent future infection. Its not a vaccine.
TBH testing is only essential when healthcare is so overwhelmed that they are placing sick patients in crowded open bay rooms. In that case you do want to want to avoid mixing positives and negatives in the same room.
Mandatory Quarantine should not require contact with tested positive. Simple Isolation should begin anytime you just have mild symptoms or known exposure to someone with symptoms.
Even treatment does not really require testing results while we lack drugs that directly fight COVID-19. TBH in current situation it does not matter if some other disease or COVID-19 is the cause when treating symptoms. If you cannot breath you still need the ventilator. Treatment follows symptom severity regardless of test results. Treatment only varies if its known that you have some other non-COVID-19 disease with specific drugs that help. Of course you can have both.
Testing is primarily useful for planning bulk resource allocation.
I think it’s a mistake. They meant to say that if the bar is too low, then a subset of patients who test positive will not have Covid-19 (i.e., specificity is decreased while sensitivity is increased).
Is this sentence worded correctly/convey what the authors intended? “But if the bar for calling a test positive is set too low, then a subset of patients who do have Covid-19 will test negative for it.”
Very well written article. It is difficult to explain statistics in simple manner as the author did. The question is what if you test only high risk or intermediate risk population? How would it affect the specificity and sensitivity? I think if one combines a simple statistical algorithm, example here https://aicovid19.com and then test the population at risk, the test sensitivity will be high and will improve specificity as well.
This is the right idea. Testing a higher-risk population doesn’t actually change the sensitivity and specificity of the test, but it does improve what’s called the positive predictive value, which is the likelihood that a positive test is a true positive (which is different from the probability that a true positive is called as a test positive, because it depends on the rate of true positives). It is a mathematical topic, but Wikipedia and other sources have good descriptions.
An important discussion with many valid points — especially regarding the importance of widespread testing.
But I think Ken and Arjun glossed over an important consideration when saying:
“Could the NBA have several thousand times the rate of Covid-19 compared to the rest of the world? Probably not.”
What makes NBA players not like regular people? Especially during the season of 82 games (October to June):
• They are constantly in airports traveling from one urban environment to another.
• Their job requires them to be in frequent, intimate contact with their teammates and competitors.
• They are interacting with literally thousands of fans on a daily basis, signing autographs, taking selfies, moving from person-to-person as quickly as possible.
• They are in the bracket of a high prevalence age group.
If just one NBA player is positive, in that hyper-connected setting, it is not surprising that many would be infected. So while I take the spread of COVID-19 very seriously, NBA players look more like a collection of Patient 31s than of a typical US resident.
Excellent point, Ross Martin. Also, regarding the Chinese paper which showed 47% false positive rate amongst asymptomatic contacts – bear in mind that the reason for its retraction could very well be political rather than scientific.