Skip to Main Content

Randomized controlled trials have long been held up as the “gold standard” of clinical research. There’s no doubt that well-designed trials are effective tools for testing a new drug, device, or other intervention. Yet much of modern medical care — perhaps most of it — is not based on randomized controlled trials and likely never will be. In this “dark matter” of clinical medicine, past practices and anecdotes all too often rule. We need to look beyond trials to improve medical care in these areas.

In a randomized controlled trial (RCT), participants are randomly assigned to receive either the treatment under investigation or, as a control, a placebo or the current standard treatment. The randomization process helps ensure that the various groups in the study are virtually identical in age, gender, socioeconomic status, and other variables. This minimizes the potential for bias and the influence of confounding factors.

Despite their strengths, RCTs have substantial limitations. They can be very expensive to run. They can take many years to complete, and even then may not last long enough to assess the long-term effect of an intervention such as vaccine immunity, or to detect rare or long-term adverse effects. Findings from RCTs may not be valid beyond the study population — a trial that included a high-risk population in order to maximize the possibility of detecting an effect, for example, may not be relevant to a low-risk population. RCTs may not be practical for population-wide interventions and often aren’t relevant for urgent health issues such as infectious disease outbreaks, for which public health decisions must be made quickly.


As I write this week in the New England Journal of Medicine, several other study types can generate data that are at least as effective as RCTs, or may be even more effective, at generating evidence for action, especially related to population-wide interventions.

The effectiveness of the nasal spray flu vaccine (also called the live attenuated vaccine) is a dramatic illustration of the limitations of RCTs. Trials suggested that the nasal spray vaccine was superior to flu shots, at least for some populations. In subsequent years, however, observational studies, including case-control studies, documented that, for reasons which are still unclear, the nasal spray wasn’t effective against the flu. That led the Advisory Committee on Immunization Practices to recommend, and the CDC to accept the recommendation, that the nasal spray flu vaccine not be used in the 2016-2017 flu season.


For some public health issues, it isn’t ethical to conduct an RCT. Take sudden infant death syndrome (SIDS). Early case-control studies suggested, but didn’t prove, that babies who sleep on their stomachs are more likely to die of SIDS than babies who sleep on their backs. It wouldn’t have been ethical to randomize some babies to stomach sleeping. A public program to implement putting children to sleep on their backs proved that this measure reduced the incidence of SIDS.

It would be difficult, if not impossible, to do an RCT of community-wide tobacco control measures. But analyses of the results of implementing tobacco control policies, such as taxes, smoke-free laws, and advertising bans, have generated robust evidence of effectiveness that could not have been accomplished through an RCT-style study.

For the several thousand rare diseases, RCTs are unlikely to be conducted due to the small number of people who have them and other logistical constraints. Detailed case studies, registries that collect information about specific conditions and diseases, and other study types can enhance understanding of a particular disease and its treatment to improve the health of affected patients.

The emerging use of “big data,” including information from electronic health records and expanded patient registries, presents new opportunities to conduct large-scale studies with many of the benefits of RCTs but without the expense. One such study used data from the Veterans Health Administration and Medicare to examine outcomes of treatment for type 2 diabetes. This study was many times larger, with much longer follow-up and lower cost, than previous RCTs comparing the effectiveness of different diabetes drugs. It clearly showed that one class of drug, the thiazolidinediones, was much more effective than another class, the sulfonylureas, in reducing hospitalization and death.

Clinical and public health decisions are almost always made with imperfect data. There is no single, best approach to obtain better information about health interventions. Evidence grading systems, policy makers, and researchers must embrace other study types in addition to RCTs. Essential steps in interpreting findings and identifying data for action include promoting transparency in study methods, ensuring standardized data collection for key outcomes, and using new approaches to improve data synthesis.

Despite the global evidence base, around the world there are often claims that “there is no evidence tobacco harms health here” or that “soda isn’t proven to drive obesity in this country.” In part, such claims can be made because some formal systems of analyzing evidence give undue weight to RCTs and inappropriately discount other types of rigorously developed evidence.

A valid ideal is “evidence-based practice,” which means implementing in clinical care and public policy interventions that are proven to work. But it’s also important, and perhaps more so, to develop “practice-based evidence,” — that is, to implement programs and rigorously document whether or not they work. That would both save lives and expand the evidence base of effective interventions.

There will always be an argument for more research and for better data. But waiting for more data is often an implicit decision not to act, or to act on the basis of past practice rather than on the best available evidence. Glorifying RCTs above other approaches, even when these other approaches may be either superior or the only practical way to get an answer, relegates patients to receiving treatments that aren’t based on the best available evidence.

An approach that uses all appropriate evidence types and builds on the existing evidence base using proven best practices is the one most likely to result in clinical and public health action that will save lives.

Tom Frieden, M.D., served as director of the Centers for Disease Control and Prevention from 2009 to 2017.

  • Good article. Another point is that given genetic diversity, we need basket studies in RCTs to sort out who is helped vs. not vs. harmed. Many things are not one size fits all when it comes to drugs, medical and dental devices, even foods and supplements.

  • Note to Paul Thompson re writing up off label recommendations. If you google me, you will see that publicizing simple remedies like baby oil to prevent swimmer’s ear is not effective unless Ophrah Winfrey recommends it.

  • to Jan Patenaude, In many therapies, the brain enters into the healing. Best to UTILIZE the placebo effect – reduce stress, explain fully how therapy works, ask patient to visualize it working, give specific ACTIONS to follow.
    Even when I detail a simple action to follow, that scientifically can’t be of value, the fact that the patient TAKES CHARGE, enters into the healing.

  • Great article. Thanks. I work with sick/hurting clients who have “been there, done that” and when I/we get them “well” with significant diet changes, some will suggest it’s “placebo effect.” Well, if they made 20 different prior attempts to get well, including other diet changes, before, why now, when they get well, does somebody want to suggest it’s placebo? Placebo didn’t work 20 times before. Sometimes the “evidence” is just obvious.

  • Unfortunately there doesn’t exist any mechanism for informing peers about off label success.
    Worse, drug companies are punished severely for daring to whisper or suggest that Dr. Jones is using their drug with outstanding success for off label use. For years I begged Johnson’s baby oil be recommended to prevent swimmer’s ear.
    We know they would have been crucified had they done so.
    The drug rep does not dare even hint that someone is having great results with off label use of his product.
    There needs to be a solution to this bar to drug information

  • unfortunately this has become typical in stat. “it clearly shows …” this is someone who got wind of ‘big data analytics’, and thought it sounded sexy and they write on it without understanding it. ‘rct’s are expensive and evidence comes from elsewhere’, evidence comes from elsewhere because rcts are expensive etc. and youre going to insinuate that observational studies provide less equivocal results than rcts?? hard, objective evidence is becoming less and less fashionable for some reason as people want to appear ‘open-minded’ ….

  • In the general monitoring and evaluation world, the saying is, “Those who insist that RCTs are the only valid test for effectiveness should consider whether they’d be willing to join the RCT for parachutes.”

Comments are closed.