Results from the first studies designed to determine how widely the coronavirus has spread in communities have started to trickle in, drawing immense attention. These studies, after all, are seen as critical indicators of when it might be safe to lift movement restrictions.

Already, though, experts are raising concerns about the validity of some of the studies and cautioning officials and the general public not to put too much weight on any one finding.

Known as serological surveys, the studies involve testing the blood of people not diagnosed with Covid-19 to determine whether they had previously been infected by the SARS-CoV-2 virus. They are important because they can flesh out the picture of how many people in any given community may have had Covid-19, even if they were unaware they were infected.

advertisement

Support STAT: If you value our coronavirus coverage, please consider making a one-time contribution to support our journalism.

These findings can help policymakers and experts gauge how vulnerable a community remains to the virus, how deadly the pathogen is, and how frequently asymptomatic or mild cases occur.

“That number, whether it turns out that we’ve had 1% of the population infected or 10 or 15% of the population infected, is going to really be crucial to the decisions we make next,” said Michael Mina, an epidemiologist at Harvard’s T.H. Chan School of Public Health.

advertisement

But in the rush to get answers, the market has been flooded with a vast number of tests, most of which haven’t been validated. Some of the serological studies have also been more rigorous than others. Epidemiologists and statisticians have criticized some of the early headline-generating surveys for how they recruited participants and analyzed the data.

More results are expected soon. Each study will be a snapshot of a community at a given time, and each will have its limitations.

“What is really important for us to be able to do to put these numbers into context is to understand how the studies were done,” said Maria Van Kerkhove, the technical lead on the pandemic for the World Health Organization, which has released a protocol for conducting serosurveys.

Below, STAT outlines how these surveys work and some of the considerations experts use to assess the strength of them.

The snapshots are starting to pile up. What do they show?

These surveys involve testing a sampling of a population for antibodies — a sign that a person was exposed to the virus and mounted an immune response — and extrapolating those findings to suggest what they mean for a community as a whole. Already, researchers have found that in particularly hard-hit places, the virus could have infected double-digit percentages of the population.

Of 200 people tested on the street in the Boston suburb of Chelsea, some 30% had been exposed to the virus. A sampling of a German town where a carnival drove the spread of the virus found 14% were positive for antibodies.

And on Thursday, New York Gov. Andrew Cuomo announced that roughly one in five people in New York City and nearly 14% statewide had antibodies. The New York estimate was based on a sampling of 3,000 grocery store shoppers.

But those double-digit findings are the exceptions, not the rule. Most of the results to date have shown just a tiny fraction of people have been exposed to the virus, including one survey released this week that found 5.5% of 760 participants in Geneva were infected. Others have found 2% to 3% of people test positive.

In a way, finding that just a small percentage of people in communities has been infected is disheartening. It’s thought — though not yet proven — that people who recover from an initial infection will be protected from another case for some amount of time. The higher the percentage of immune people in an area, the less well the virus spreads. Even a third of people being protected from contracting and transmitting the virus does not mean that community is close to reaching “herd immunity” — when enough people are protected that the virus can’t gain a foothold.

“There’s been an expectation that herd immunity may have been achieved and that the majority of people in society may already have developed antibodies,” said Mike Ryan, the head of the WHO’s emergencies program. “I think the general evidence is pointing against that and pointing towards a much lower seroprevalence.”

Experts say it is imperative in these early days to review the methodologies of these studies and assess the performance of the antibody tests being used. It’s also important to recognize that many have not yet gone through the rigorous peer review process before results are made public. Some are publicized in press releases, others on preprint servers, which post drafts of scientific papers before publication in journals so that they can be shared rapidly.

“I have to admit, I am struggling with this in this pandemic,” Van Kerkhove said of the tsunami of papers being released as preprints. She applauded the rapid sharing in general, but said a lot of the papers really need the critiquing of peer review to sharpen findings and correct errors. “By the time a paper is actually published, what it looks like from the time of submission is drastically different. And that is hard to interpret.”

How do scientists recruit for their studies?

The ideal patient sample in one of these surveys will be a random one that reflects a community’s socioeconomic, geographic, age, and ethnic diversity.

One of the issues outside experts raised with a widely criticized study out of Santa Clara County, Calif., last week was that the researchers recruited participants through Facebook ads. They say this approach could have drawn people who thought they had been infected and wanted confirmation — and wouldn’t necessarily lead to a sampling that represented the county’s population as a whole. Instead, it could have resulted in infection rates that were misleadingly high.

“It biases your sample very much toward people who want to be tested, who might suspect they’ve had it, and that can lead you to overestimate the number of people who have actually been exposed,” said William Hanage, an epidemiologist at Harvard’s T.H. Chan School of Public Health.

To recruit a representative sampling for a survey in Florida’s Miami-Dade County, the research team used information from the local power utility to randomly dial residents with a recorded call from the mayor to ask if they would enroll.

Even this approach is not perfect, said Erin Kobetz, a professor of medicine and public health sciences at University of Miami, who is helping run the survey. Some people, for example, won’t have transportation to get to the testing site or will have other reasons to participate or not. She said researchers needed to acknowledge the limitations of their studies.

“This is important information for public health planning, because it’s telling us what’s occurring at a population level,” she said. “Is it perfect? No. Is it better than nothing? Yes.”

Hanage said that conducting a survey with more participants and in a community that has had wider spread of the virus will produce more reliable results. Greater exposure makes it more likely the random sample will include people who have had serious illness as well as mild or asymptomatic illness, so the survey can “determine the real spectrum of severity of disease,” he said. Otherwise, the flaws in the tests in producing false positives or false negatives will be exacerbated. (More on that later).

Do the results pass the smell test?

Far more people have been infected by the coronavirus than official counts reflect. Given the range of symptoms, many people never felt sick enough to seek out a test, and many who did couldn’t get one. In countries including the United States, testing has been limited by problems with the rollout and capacity.

But when the team that led the Santa Clara County study reported that the case count could be 50 to 85 times higher than the number of confirmed cases, that raised skepticism. If that were truly the case, then local hospitals would have been far more inundated with patients than they had been, outside researchers argued. Other estimates have pegged total cases as 10 to 20 times higher than confirmed cases.

Similarly, that higher infection rate would drive the fatality rate (which is established by dividing deaths over total infections) far lower than what’s been seen throughout the pandemic. Making a claim that goes against so much other data requires a ton of evidence that outside experts said this study did not provide.

So about those false positives and false negatives?

No test is perfect. And the sheer number of antibody tests — Dutch virologist Marion Koopmans recently saw nearly 275 on a list maintained by the WHO — makes it very tough at this stage to know how good any of them actually are. The WHO is working with a number of labs trying to validate tests, said Van Kerkhove, who added: “Unfortunately that takes a little bit of time.”

In particular, the rapid tests appear not to perform well at all. Koopmans, the head of virology at Erasmus Medical Center in Rotterdam, said the Dutch national serology task force has recommended that people not use the rapid tests, because of the risk that people will get a false result and assume — if it was a positive — that they have protection they do not in fact have.

Every serology test is going to produce some erroneous results. Some people who were truly sick will test negative — that’s a false negative. Some people who were not sick will test positive — that’s a false positive.

Each commercial test comes with guidance from the manufacturer about how “sensitive” it is — in other words, what percentage of true positive cases it will detect — as well as how “specific” it is, meaning how good it is at not generating false positive results.

Those estimates are especially important when the rate of infection in an area is likely low. Even a small over-estimate — say a 5% false positive rate — can vastly increase the final projection of how many people in a location had been infected.

Michael Osterholm, director of the Center for Infectious Diseases Research and Policy at the University of Minnesota, drew up a chart to explain how different rates of sensitivity and specificity will impact a serology study in an area with 1 million people, using a test that had 95% sensitivity (caught all but 5% of true positives) and 95% specificity (designated as positive only 5% of people who were actually negative).

If 5% of the population had been infected with SARS-CoV-2, there would have been 50,000 infected people. This test would find 47,500 (the true positives) but it would miss 2,500 (the false negatives). And it would detect 47,500 false positives — as many false positives as true positives. If the rate of infection in the community was smaller, the percentage of wrong results would rise.

If the rate of infection in the community increased, the errors become less substantial. If 15% of the community — 150,000 — had been infected, this test would find 142,500 true positives, 42,500 false positives, and would miss 7,500 cases — the false negatives.

Applying this knowledge to Thursday’s results from New York puts the picture in sharper focus. The release from the state doesn’t disclose the sensitivity of the test used, but it does note the specificity is between 93% and 100%, a “huge range,” Ashish Jha, head of Harvard’s Global Health Institute, noted on Twitter. If the test performed at the low end of that range, New York’s infection rate would be closer to 7% — half the figure Cuomo announced — and nearly one out of every two positives would have been a false positive, Jha said.

“These tests don’t perform like people think they do and so there are a lot of crazy results,” Osterholm said. “You can often find more than half of the positives you do document are actually false positives.”

People who don’t understand how challenging serology testing is may assume a result is binary — positive or negative. But reading a result is nowhere near that black and white, Osterholm said.

Think that’s complicated? There’s more.

The whole goal with serological surveys is to get an estimate of the proportion of cases that testing missed. That means finding the people with mild infection.

But some emerging evidence suggests at least some people with really mild or almost symptom-free infections may have very low levels of antibodies — or no detectable antibody at all. That will complicate both the testing and the interpretation of any results.

“If people with mild disease don’t have antibodies, do they then have some kind of immunity or not at all?” Koopmans wondered.

Another potential complication arises from the fact that SARS-CoV-2 is a member of the coronavirus family, which includes four viruses that cause common colds. Most of us would have been infected by at least a few of these four over the courses of our lifetimes.

Are the new serology tests good enough to distinguish SARS-2 antibodies from those generated by the other human coronaviruses? Or will some of them pick up a false signal?

“We know that it’s a potential problem,” the WHO’s Van Kerkhove said, adding that’s why it is so important that tests are validated. Part of the validation process is testing the assays on blood samples drawn — before the new virus emerged — from people who have had previous coronavirus infections.

In the United States, the Food and Drug Administration has taken the extraordinary step of allowing test manufacturers to market serological tests that haven’t yet undergone agency review, as long as they have undergone validation in the hands of the manufacturer.

But Koopmans said sometimes this type of validation is minimal.

Natalie Dean, an assistant professor of biostatistics at the University of Florida who has been critiquing some of the serology studies on Twitter, said this approach is kind of an about-face for the FDA.

“I guess one of the important things to remember is that the FDA faced all those challenges with PCR testing” — the type of testing used to confirm active infection — “and being very stringent. And so what appears to have happened is sort of a swing in the other direction, where now they’re being quite lax,” Dean said.

Still, with all the current problems, two things are clear.

The first: Whether the study was conducted in California or Denmark, in the Netherlands or Germany, most have shown the virus has not yet infected a big portion of the screened population.

“In general I think things have been pretty consistent. It’s only in the really hard-hit places that we’re seeing anything above a single-digit number,” said Dean. “The harder hit places have higher seroprevalence, but even the hard hit places don’t seem to have crazy numbers. Certainly not at the herd immunity level.”

That leads us to the second thing: The world still has no idea how to interpret the import of any of these studies.

The presence of antibodies suggests a person was previously infected. But are they protected? Most experts assume there will be at least short-term protection. But how long will it last? Will people with low levels of antibodies be as protected as people with higher levels? Could countries safely issue “immunity passports,” as several have suggested, that would allow people with proof of prior infection to return to work or move about more freely?

Only more experience with this new virus will provide answers to those kinds of questions. “At the current time, there’s a lot of problems with that [idea],” Van Kerkhove said about the immunity passports. “Because antibody does not mean immunity.”

  • Helen – I trust you to get to the bottom of this. What everyone in the United States wants to know: why the logjam in testing? Realistically, how long will it take to get to the widespread testing called for under most reopening guidelines? How long to train the necessary and deploy the necessary contact tracers? Are we talking weeks? Months? Many months? And what do state governments intend to do if the answer is several months? And lastly, is contact tracing even feasible in a country of our size, with open state borders, an open southern border and an estimated 10 million undocumented workers who will not want to be traced?

    I greatly respect your reporting abilities. Please, Americans desperately want these answers and it feels like we are being evaded, patronized and perhaps even lied to.

    • We still don’t know how many people have been infected with the novel coronavirus, SARS-CoV-2. Not only have countries struggled to roll out wide-scale testing for the virus, those efforts inevitably will miss people who have recovered from an infection. The best way to figure out how far and wide the virus has spread in a population is to look at blood. Antibodies, blood proteins that the immune system produces to attack pathogens, are viral fingerprints that remain long after infections are cleared. Sensitive tests can detect them even in people who never felt a single symptom of COVID-19.
      The World Health Organization has announced an ambitious global effort, called Solidarity II, of so-called serosurveys, studies that look for antibodies to SARS-CoV-2 in the population. There’s a whole portfolio of “use cases” for serologic testing. You can use serology as an adjunct to the swab testing that detects the virus to help diagnose acute infections. It can identify donors of plasma from recovered patients that can be transfused into COVID-19 patients as a potential treatment. It can help estimate the timing of infection. And for vaccines, we need to have serologic tools that can discriminate between vaccine-induced antibodies and natural infection.The proportion of people who have recently acquired SARS-CoV-2 who would be positive with a single time point with nasal pharyngeal swab—the usual diagnostic sample, which uses the polymerase chain reaction to amplify tiny bits of viral nucleic acid so it can be detected—is probably 50%, or at best 70% to 80%. Antibody testing wouldn’t pick people up in the earliest stages of infection who had asymptomatic infections, but the data are becoming very solid that within 4 to 5 days of earliest disease onset, antibodies are detectable. So if you really want to pick up acute infections, you need to add serology to nucleic acid testing.

  • “Even a third of people being protected from contracting and transmitting the virus does not mean that community is close to reaching ‘herd immunity'”
    Well that depends on the infection rate [Basic Reproduction Number, R 0]. With a low infection rate, even a 33% herd immunity could stifle an epidemic. With a high infection rate, a much higher herd immunity percentage is needed.

  • There are separate issues that need to be rigorously separated.
    The background exposure will tell us the case fatality rate. All evidence points to a far lower case fatality rate than that promoted at this point; and at some cases, not out of the neighborhood of a really bad flu season.

    If that is true, all the damage imposed by economic shutdown is an immense waste of resources.

    The second is the natural herd immunity rate. This is essential to prevent the re-emergence of the virus. Our current shutdown is clearly preventing herd immunity from emerging so we are in a circular logic: Insufficient herd immunity argues for the shutdown; but the shut down is preventing the emergence of herd immunity.
    If a vaccine can be made (and this is a big “IF” with an open time frame, herd immunity can be induced on the population

    Some logical reasoning is needed by both politicians and medical opinion makers.

    • All evidence? New York State’s serological survey points to a case fatality rate of 0.76%, and it could be twice that depending on the accuracy of the test. There is also a high likelihood of thousands of undocumented coronavirus deaths, based on unexplained excess mortality. This could still push the 0.76% estimate over 1%. South Korea’s extensive testing and contact tracing shows a case fatality rate of 2.2%.
      We also don’t know how long herd immunity could last, or if it’s even possible. SARS and MERS antibodies are present for years, but common cold coronavirus antibodies only last an average of three months.

  • I’m curious about the possible interference of Biotin with these rapid antibody assays. Troponins, various hormones, antibodies to Hep A, Hep B, and Hep C, among other tests can all show false-negative results if the patient has consumed Biotin. I’ve contacted some manufacturers and studies about this concern but have received no response. If it interferes with the test, they need to make people aware of this and not test patients until two weeks after they’ve stopped supplementing.

    • Wow thank you for that. Amazing that the FDA has known since 2017 yet the news is not widely known. Hairdressers recommend biotin all the time.

  • One significant problem with immunity certificates relates to zhe issue brought up by Michael Osterholm in this article. That is the large number of false positives in populations with low prevalence even when the test itself has a validated high specificity. The result is that you may be handing out certificates based on the eqivalent of a coin toss or worse (50% or less chance of a positve for antibodies being a true positive). These certified, but still very much susceptible individuals, will not only be potentially more exposed, but also will take fewer precautions because they presume immunity. The makings of a disaster. It is difficult to understand why WHO and other public health authorities are not making this issue more clear to all those wanting to use antibody tests to establish individual immunity.

    • I think any “immunity certificates” will have to be based on at least two positive tests from different vendors.

      I also expect that we will be seeing more seroprevalence studies where they validate their test against two or more tests from other vendors, because it is so important to nail down the test error rates to narrow the error bars on the prevalence estimates.

    • “The result is that you may be handing out certificates based on the eqivalent of a coin toss or worse (50% or less chance of a positve for antibodies being a true positive)”

      Sorry, that is not a coin toss. If you consider a prevelence rate of 5% and false positive rate of 5% (making close to 50% change of positive being a true positive), a coin toss is going to only give 5% chance of positive being a true positive. Besides, the current antibody tests seem to have only about 0.5% – 1% false positive rate. And lastly as someone else mentioned, if you require two positive results, the stats become much better.

  • Why are studies or reports about coronavirus being less lethal or less deadly than initial estimates nearly universally met with skepticism?

    We created our public policy responses based on doomsday scenarios that have not played out, and now it seems like many experts are more invested in seeing those scenarios be right than admitting they were working with faulty data.

    Reminds me of that “the sky isn’t falling” article the very respectable Dr. Ioannidis wrote here at the beginning of the crisis that triggered nasty (and unwarranted) recriminations.

    • Why do people keep chiming in with similarly worded comments that fail to respond to major problems with the Santa Clara study?

  • I am interested in these studies partly to determine the true case fatality rate, right now, in the US, particularly Santa Clara County where one of these highly publicized studies was done.

    All the debate, and, seems to me, wishful thinking, about unidentified infected people has confused the issue.

    Can anyone point me to a study of patients identified as infected by CoV2, that is, actually diagnosed, and how they have done so far? It is not clear to me it is not much higher than the roughly 3% we are often told.

    With the epidemic 6 weeks old now, there ought to be some meaningful numbers by now.

    • The Diamond Princess study showed a 0.5% mortality rate, but that included data from China, which may very well render it meaningless. South Korea’s numbers are likely fairly accurate, given the aggressive testing and lack of continued spread. They show a 2.2 case fatality rate, but that can’t be directly translated to other populations with different age, general health and prevalence of underlying conditions, etc.

  • Compared to the fawning acceptance of early doomsday models that turned out — in the real world — to be by off by orders of magnitude, the extreme skepticism applied to actual data collected in the real world is telling. It would be very hard — unimaginably so — for an expert to admit that he or she made mistakes that led to devastating and unwarranted policy decisions. That bias should be strongly considered when allowing experts to tell us how to think about this important, increasing, and echoing body of information.

    • “Early doomsday models” were never tested vs reality, because of varying degrees of public health measures. New York’s statistically meaningful (unlike the Santa Clara data) serology study study shows possibly a 0.76% case fatality rate, but 2.5 – 3% is not outside the range of uncertainty.

    • We are far too early in the course of this disease to know when or if it will be a doomsday event.

  • With the conclusive irrefutable positive Covid 19 test of a woman who deceased on Feb 5th in Santa Clara CA with no travel history, there is evidence of a substantial earlier outbreak. Because now the antibody science has been politicized and scientists are rushing to criticize or support studies that back their bias, perhaps to scare people into adhering to sheltering (while most are doing so regardless), theIr lack of openness and curiosity is sowing the seeds of mistrust of science in the media. This is the same mistrust of our federal administration’s information, that can eventually lead to chaos.

    • That’s evidence of 50 – 200 other cases In late January, assuming the fatality rate is somewhere between 0.5 to 2%.

  • Why wasn’t the same degree of skepticism applied to the initial alarming estimates of millions of deaths? All of those have been revised down or retracted entirely? None of those passed peer review either. “Epidemiologists and statisticians have criticized some of the early headline-generating surveys” — that quote refers to the antibody tests. But it applies just as much to the COVID death estimates. Yet, those early estimates were used to justify the draconian shut-downs.

    • I agree. With early serology test data coming in, the death rate may be 0.2%, roughly the same as influenza (0.15%), and affecting the same population. So our scientists have politicized their projections at great cost to the lives of our citizens. Think about it, nobody invents a ‘model’ without strong incentive to gain attention for it by claiming the sky is falling. We are the suckers for this.

    • The models are revised because people changed their behavior. Without social distancing and 60% of the population needed to achieve herd immunity, that’s 200,000,000 cases in the US. A low-end estimate of 0.5% fatality leaves one million people dead.

      Here’s a good write-up of why the Santa Clara study doesn’t show meaningful results, in addition to contradicting common sense and observation:

      https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanford-study-of-coronavirus-prevalence/

    • The Centre for Evidence-Based Medicine at the University of Oxford gives the range of Infection Fatality Rates, as of 24th April 2020, between 0.1% and 0.36%.

      So no, 0.5% is not the “low-end estimate”.

      They admit “antibody testing will provide an accurate understanding of how many people have been infected so far, and permit a more accurate estimate of the IFR.” So check back later – maybe the range will go lower.

      At 200 million infections (even though the Spanish flu 0f 1918-1919 infected only 28% of the USA, and the Swine flu of 2009-2010 only 24%) with an IFR of 0.1%, that’s 200,000 deaths – about twice the high-end estimate of deaths due to the bad flu season of 2017-2018.

Comments are closed.

Sign up to receive a free weekly opinions recap from our community of experts.
Privacy Policy