If your doctor diagnoses you with chronic fatigue syndrome, you’ll probably get two pieces of advice: Go to a psychotherapist and get some exercise. Your doctor might tell you that either of those treatments will give you a 60 percent chance of getting better and a 20 percent chance of recovering outright. After all, that’s what researchers concluded in a 2011 study published in the prestigious medical journal the Lancet, along with later analyses.

Problem is, the study was bad science.

And we’re now finding out exactly how bad.

advertisement

Under court order, the study’s authors for the first time released their raw data earlier this month. Patients and independent scientists collaborated to analyze it and posted their findings Wednesday on Virology Blog, a site hosted by Columbia microbiology professor Vincent Racaniello.

The analysis shows that if you’re already getting standard medical care, your chances of being helped by the treatments are, at best, 10 percent. And your chances of recovery? Nearly nil.

The new findings are the result of a five-year battle that chronic fatigue syndrome patients — me among them — have waged to review the actual data underlying that $8 million study. It was a battle that, until a year ago, seemed nearly hopeless.

When the Lancet study, nicknamed the PACE trial, first came out, its inflated claims made headlines around the world. “Got ME? Just get out and exercise, say scientists,” wrote the Independent, using the acronym for the international name of the disease, myalgic encephalomyelitis. (Federal agencies now call it ME/CFS.) The findings went on to influence treatment recommendations from the CDC, the Mayo Clinic, Kaiser, the British National Institute for Health and Care Excellence, and more.

But patients like me were immediately skeptical, because the results contradicted the fundamental experience of our illness: The hallmark of ME/CFS is that even mild exertion can increase all the other symptoms of the disease, including not just profound fatigue but also cognitive deficits, difficulties with blood pressure regulation, unrestorative sleep, and neurological and immune dysfunction, among others.

Soon after I was diagnosed in 2006, I figured out that I had to rest the moment I thought, “I’m a little tired.” Otherwise, I would likely be semi-paralyzed and barely able to walk the next day.

The researchers argued that patients like me, who felt sicker after exercise, simply hadn’t built their activity up carefully enough. Start low, build slowly but steadily, and get professional guidance, they advised. But I’d seen how swimming for five minutes could sometimes leave me bedbound, even if I’d swum for 10 minutes without difficulty the day before. Instead of trying to continually increase my exercise, I’d learned to focus on staying within my ever-changing limits — an approach the researchers said was all wrong.

A disease ‘all in my head’?

The psychotherapy claim also made me skeptical. Talking with my therapist had helped keep me from losing my mind, but it hadn’t kept me from losing my health. Furthermore, the researchers weren’t recommending ordinary psychotherapy — they were recommending a form of cognitive behavior therapy that challenges patients’ beliefs that they have a physiological illness limiting their ability to exercise. Instead, the therapist advises, patients need only to become more active and ignore their symptoms to fully recover.

In other words, while the illness might have been triggered by a virus or other physiological stressor, the problem was pretty much all in our heads.

By contrast, in the American research community, no serious researchers were expressing doubts about the organic basis for the illness. Immunologists found clear patterns in the immune system, and exercise physiologists were seeing highly unusual physiological changes in ME/CFS patients after exercise.

I knew that the right forms of psychotherapy and careful exercise could help patients cope, and I would have been thrilled if they could have cured me. The problem was that, so far as I could tell, it just wasn’t true.

A deeply flawed study

Still, I’m a science writer. I respect and value science. So the PACE trial left me befuddled: It seemed like a great study — big, controlled, peer-reviewed — but I couldn’t reconcile the results with my own experience.

So I and many other patients dug into the science. And almost immediately we saw enormous problems.

Before the trial of 641 patients began, the researchers had announced their standards for success — that is, what “improvement” and “recovery” meant in statistically measurable terms. To be considered recovered, participants had to meet established thresholds on self-assessments of fatigue and physical function, and they had to say they felt much better overall.

But after the unblinded trial started, the researchers weakened all these standards, by a lot. Their revised definition of “recovery” was so loose that patients could get worse over the course of the trial on both fatigue and physical function and still be considered “recovered.” The threshold for physical function was so low that an average 80-year-old would exceed it.

In addition, the only evidence the researchers had that patients felt better was that patients said so. They found no significant improvement on any of their objective measures, such as how many patients got back to work, how many got off welfare, or their level of fitness.

But the subjective reports from patients seemed suspect to me. I imagined myself as a participant: I come in and I’m asked to rate my symptoms. Then, I’m repeatedly told over a year of treatment that I need to pay less attention to my symptoms. Then I’m asked to rate my symptoms again. Mightn’t I say they’re a bit better — even if I still feel terrible — in order to do what I’m told, please my therapist, and convince myself I haven’t wasted a year’s effort?

Many patients worked to bring these flaws to light: They wrote blogs; they contacted the press; they successfully submitted carefully argued letters and commentaries to leading medical journals. They even published papers in peer-reviewed scientific journals.

They also filed Freedom of Information Act requests to gain access to the trial data from Queen Mary University of London, the university where the lead researcher worked. The university denied most of these, some on the grounds that they were “vexatious.”

Critics painted as unhinged

The study’s defenders painted critics as unhinged crusaders who were impeding progress for the estimated 30 million ME/CFS patients around the world. For example, Richard Horton, the editor of the Lancet, described the trial’s critics as “a fairly small, but highly organised, very vocal and very damaging group of individuals who have, I would say, actually hijacked this agenda and distorted the debate so that it actually harms the overwhelming majority of patients.”

Press reports also alleged that ME/CFS researchers had received death threats, and they lumped the PACE critics in with the purported crazies.

While grieving for my fellow patients, I seethed at both the scientists and the journalists who refused to examine the trial closely. I could only hope that, eventually, PACE would drown under a slowly rising tide of good science, even if the scientific community never recognized its enormous problems.

But with the National Institutes of Health only funding $5 million a year of research into chronic fatigue syndrome, it seemed like that could take a very long time.

Then last October, David Tuller, a lecturer in public health and journalism at the University of California, Berkeley, wrote in Virology Blog a devastating expose of the scientific flaws of the trial. Tuller described all the problems I had seen, along with several more. The project was a remarkable act of public service: He isn’t a patient, yet he spent a year investigating the trial without institutional support, legal backing, or remuneration.

And, at last, the criticisms gained traction.

Racaniello and 41 other scientists and clinicians published an open letter to the Lancet calling for an independent investigation into the trial and saying “such flaws have no place in published research.” Rebecca Goldin, the director of Stats.org, an organization that works to improve the use of statistics in journalism, eviscerated the trial’s design in a 7,000-word critique.

In the meantime, a Freedom of Information Act request from Australian patient Alem Matthees was making its way through the legal system.

Matthees had asked for the anonymized data necessary to analyze the study using its original standards for success, but Queen Mary University of London had refused the request, arguing that malicious patients would break the anonymization and publish the participants’ names to discredit the trial. It again cited the death threats.

The court rejected these claims a month ago, calling them “wild speculations” and pointing out that the researchers themselves acknowledged in court that neither they nor PACE participants had received death threats.

Startling results from a re-analysis

Just before releasing the data,Queen Mary University of London did its own re-analysis on the question of how many patients got better, at least a little bit. Their data showed that using the study’s original standards, only 20 percent of patients improved with cognitive behavior therapy or exercise in addition to medical care, not 60 percent as claimed in the Lancet.

And even the 20 percent figure might be misleading, because the re-analysis also found that 10 percent of participants improved after receiving only standard medical care. That suggests that 10 percent in each of the treatment groups would likely have improved even without the exercise or therapy, leaving only 10 percent who were significantly helped by those interventions.

As for the claim that 22 percent of patients who received either treatment made an actual recovery? That went up in smoke when Matthees analyzed the raw data with the help of his colleagues and statisticians Philip Stark of the University of California, Berkeley, and Bruce Levin of Columbia University.

Their analysis showed that had the researchers stuck to their original standards, only 4.4 percent of the exercise patients and 6.8 percent of the cognitive behavior therapy patients would have qualified as having recovered, along with 3.1 percent of patients in a trial arm that received neither therapy.

Importantly, there was no statistically significant difference between these recovery rates.

The PACE researchers, the editor of the Lancet, and the editors of Psychological Medicine (which published the follow-up study on recovery) all declined to comment for this article.

Simon Wessely, president of the UK Royal College of Psychiatrists, defended the trial in an email exchange with me. He argued that some patients did improve with the help of cognitive behavior therapy or exercise, and noted that the improvement data, unlike the recovery data, was statistically significant. “The message remains unchanged,” he wrote, calling both treatments “modestly effective.”

Wessely declined to comment on the lack of recovery. He summarized his overall reaction to the new analysis this way: “OK folks, nothing to see here, move along please.”

‘A classic bad study’

But it doesn’t appear that outside researchers are ready to “move along.”

After reviewing the new analysis, Jonathan Edwards, a professor emeritus of medicine at University College London said he was unconvinced that these small subjective improvements indicated the patients genuinely felt better. “They’ve set this trial up to give the strongest possible chance of there being a placebo effect that you can imagine,” he said.

“This is a classic bad study,” said Ron Davis, director of the Stanford Genome Technology Center and director of the Science Advisory Board of the End ME/CFS Project. He emphasized an additional problem: The study used such a broad definition of the disease that it likely included many patients who didn’t truly have ME/CFS at all.

“The study needs to be retracted,” Davis said. “I would like to use it as a teaching tool, to have medical students read it and ask them, ‘How many things can you find wrong with this study?’”

Retractions are rare, however, and erasing the impact of this flawed research will take much work for years to come.

After a sustained effort by ME/CFS advocates, the federal Agency for Healthcare Research and Quality, just changed its recommendation to read that there is insufficient evidence to justify cognitive behavior therapy or graded exercise. But many more public health agencies continue to point patients toward them.

And efforts to propagate this approach continue: A trial of graded exercise in children with ME/CFS has recently begun, and patients are protesting it.

Watching the PACE trial saga has left me both more wary of science and more in love with it. Its misuse has inflicted damage on millions of ME/CFS patients around the world, by promoting ineffectual and possibly harmful treatments and by feeding the idea that the illness is largely psychological. At the same time, science has been the essential tool to repair the problem.

But we shouldn’t take solace in the comforting notion that science is self-correcting. Many people, including many very sick people, had to invest immense effort and withstand vitriol to use science to correct these mistakes. And even that might not have been enough without Tuller’s rather heroic investigation. We do not currently have a sustainable, reliable method of overturning flawed research.

And rectifying PACE will take more than exposing its flaws. The lingering doubt it has cast on the illness will only be fully dispersed when we’ve finally figured out what’s really going on with the disease.

For that, we need to invest in some serious, good science. The kind I continue to love.

Julie Rehmeyer is a math and science writer. Her memoir “Through the Shadowlands,” describing the science and politics of chronic fatigue syndrome and other poorly understood illnesses, will be published by Rodale in May.

Leave a Comment

Please enter your name.
Please enter a comment.

  • Excellent read! The unfolding of this story highlights human and institutional fallability in science – such as in rationalizing a dogged pursuit to save face.

  • CFS/ME fix
    1. Detect the pathogenic source
    2. Remove the pathogenic source
    3. Reboot the body control systems.
    Reserch cannot solve the CFS. Treatment is only possible if you have the superpower to look for an invisible cause. Science can’t study superpowers yet.

  • Bless you. I’m printing this out to hand to every doctor that ever had or ever will tell me I need to do “graded exercise” without being able to explain exactly what that entailed and despite my dermatologist report that says sweating gives me hives. Swimming laps makes me happy but gave me eczema at 55. I went out for groceries today. I likely will be in bed for the next two days.

    • Interesting you mentioned eczema. Prior to various physical issues I was a swimmer, 72 laps at least 5 days a week. I also had bad eczema. One doctor called it a type of eczema that in medical terms means “allergic to water”, I don’t remember the word.
      I rarely swim now and although obviously water is on my body it’s not immersed as much as with swimming and I’ve not had an eczema breakout for a couple years.
      Perhaps something to ask your doctor

  • Brian Hughes in his book Psychology in Crisis – of which I’ve only managed to read the chapter he devotes to ME/CFS – IIUC suggests serious problems like in the PACE Trial are endemic in psychology research; that it often uses very low grade science that as you describe, has preconceived outcomes with trials set up with only one possible outcome allowed – to prove the authors’ hypothesis, and that this situation has somehow gone unchallenged for who knows how long, by top academic institutions, journal editors, peer reviewers and Cochrane reviews.
    It’s possible that for anyone involved in this situation to admit the PACE Trial along with other psych based exercise trials and their reviews have been so severely flawed, would in effect be to admit that very poor science by psychologists, that would not be accepted in any other branch of science, has been accepted by them many times in the past, not just in ME/CFS related research.
    It would be an Emperor’s New Clothes moment that would be in effect admitting the inadmissible – their own history of lack of rigor and scientific scruples. Since their whole stock-in-trade is their high professional reputation and the level of trust put in them, any such admission could destroy them and/or their organisations.
    I tend to think these people are not so much money motivated as motivated by defending their beliefs and reputation. I think they believe they’re right and that they’re genuinely trying to help, and simply don’t understand why we’re not all more grateful. There’s a cognitive dissonance issue that doesn’t allow them to consider the reality that their fundamental belief system is deeply flawed and the careers some have pursued for decades, have been worthless, and done great harm.
    I agree that a thorough high level investigation of this whole issue is warranted, but it has to be done by someone with a lot of official influence in government and health/ medicine yet who’s not themselves part of the mutually supporting old boys’ club problem.
    I’m sorry this is not better argued/ expressed – battling brain fog…

  • Great job from all involved in debunking this bad science. What troubles me however is that this PACE study seemed to be intentionally constructed in a way as to reach a predetermined conclusion. What were the intentions of the PACE researchers, Queen Mary University of London and the Lancet? What motivated them in this?

    Scientists have taken this as far as they can – I would love for someone else to pick this up and delve deeper into this saga, perhaps starting by following the funding trail.

  • Wow, what an infuriating but inspiring story! Just unbelievable how hard patients need to work to get respect, help, or even just justice. You might be interested in this cartoon tribute to the patients who fought to expose the PACE Trial flaws: https://www.doodlethru.com/217

    Thanks for your article,
    Jill

  • @Katharine Keeling Spann, your story breaks my heart. I so wish that you could find useful help. As to the timing, right after Sharpe’s announcement that he will leave the field, after publishing more of his harmful nonsense, I hope he does leave. He has proved to be unable to accept that his expensive study did not show what he expected, meaning that he needed to change direction and follow the facts. Instead he is blind to the damage of GET, which is what you tried under a doctor’s direction. When I could work, we ran into a similar situation, our hypothesis was not supported by the data we collected from people, but the difference is that we tried another hypothesis that did work. My field was transportation, not medicine – – why were our standards higher than those in psych research?

  • I have suffered with ME/CFS for 25 years. Several years ago, a new doctor (specializing in ME/CFS & Fibro) told me that my deconditioned state was what was making me so fatigued….not the other way around. So I began his graded exercise program with a therapist who came to my home. After the first appt. I was so exhausted and in pain. By the third visit, I really thought I couldn’t make it through the 20 min session of arm windmills, leg lifts, and arm stretching. But I knew if I quit-my doctor would say I wasn’t trying, etc. The fourth time the therapist came I literally couldn’t roll over in bed. I cried out when they tried helping me sit on the side of the bed. I began to feel something between fainting and a panic attack. They took my pulse and it was 135. My blood pressure was going crazy. The therapist immediately laid me back in bed and said he believed from the 2nd appt that I had been getting worse. I never saw the therapist or that doctor again.
    From that one incident I became bedridden. It was over the course of 3 months ; but just standing to brush my teeth was too much. All specialists of any kind are over 90 miles away. And since we can’t afford an ambulance to take me to and from appointments-I have been unable to receive any healthcare.
    PACE was more than bad science. It was outright harmful! Post exertional malaise is the hallmark of ME/CFS. What did they think would happen?
    Thanks, Katharine
    (Typed by spouse over 2 days)

  • On why graded exeercise doesn’t usually help, why “…swimming for five minutes could sometimes leave me bedbound, even if I’d swum for 10 minutes without difficulty the day before…”, and that ME may be related to fluoroquinolone toxicity effects, I have a thought or two I’d like to share with you. However, not being a scientist or researcher, I don’t know the best way, or if you’d welcome hearing from me. If interested, please let me know. I wouldn’t share in an open place, since the thoughts derive from recent FQAD-related personal health experiences. Ty.

  • I wish someone had contacted me. Dx’ed with CFS around 2000 when I was a Personal Trainer and Fitness Instructor I was asked by an endocrinologist how I could teach aerobic classes if I was so tired. I responded “not well”.
    I was also pursuing a degree in psychology and knew about CBT, tried it and felt it wasn’t an effective therapy for me. I still believe CBT can be effective for some and not others, condition not being the factor.
    I was brought to tears as I read your article. The doctor (researcher ?) who stated that some who were dx’ed with CFS may have been misdiagnosed; today I am physically disabled with a motor neuron disorder doctors have yet to decide what to call. The closest they’ve agreed on is hEDS, I think because I take the Beighton scale to new levels and could long before fitness was a part of my life. Can’t everyone rest their forearms on the floor when they stand up straight?
    I wonder if one doctor had taken my concerns seriously; one doctor hadn’t patted my head and dismissed me with “oh sweetie it’s just stress”; just one doctor…
    I know doctors spend many years studying and learning. I have a son in med school now and I rarely see him because he is always studying for the next big test.
    That said, I know my body better than any doctor can memorize words in a textbook. Can doctors decide to be partners in health and take both banks of knowledge and we work as a team to help find answers?
    My oldest son is afflicted with schizophrenia (one reason some doctors dismiss my illness today, I’m just “stressed” about my son. News flash, I’ve also learned how to work through my stress). In the early stage of his illness during a lucid moment he angrily stopped me and said, “Mom, you study this! Why didn’t you know this was happening?”
    Because what they teach you in textbooks is completely different than the real world.
    Thank you for exploring this, for writing this article and for sharing your gift.
    Cindy Rumlow

    • Cindy, have you ever considered mold biotoxin illness/CIRS? The original CFS patient, Erik Johnson, figured out that mold is behind CFS. My daughter suffers from CIRS and mold avoidance is saving her life. Your own story correlates very closely with many in the mold groups I’m in. Check out the website Surviving Mold, get a naturopath to order the urine mycotoxin test from Great Plains Labs, and please consider reading “break the mold” by Dr. Jill Crista as a great introduction, and then “Toxic” by Dr. Neil Nathan as a very detailed follow-up. Thankfully, many more are waking up to the dangers of mold – it might even be behind your son’s schizophrenia. Please, please read up on it and look into it.

A roundup of STAT’s top stories of the day in science and medicine

Privacy Policy