I

f your doctor diagnoses you with chronic fatigue syndrome, you’ll probably get two pieces of advice: Go to a psychotherapist and get some exercise. Your doctor might tell you that either of those treatments will give you a 60 percent chance of getting better and a 20 percent chance of recovering outright. After all, that’s what researchers concluded in a 2011 study published in the prestigious medical journal the Lancet, along with later analyses.

Problem is, the study was bad science.

And we’re now finding out exactly how bad.

advertisement

Under court order, the study’s authors for the first time released their raw data earlier this month. Patients and independent scientists collaborated to analyze it and posted their findings Wednesday on Virology Blog, a site hosted by Columbia microbiology professor Vincent Racaniello.

The analysis shows that if you’re already getting standard medical care, your chances of being helped by the treatments are, at best, 10 percent. And your chances of recovery? Nearly nil.

The new findings are the result of a five-year battle that chronic fatigue syndrome patients — me among them — have waged to review the actual data underlying that $8 million study. It was a battle that, until a year ago, seemed nearly hopeless.

When the Lancet study, nicknamed the PACE trial, first came out, its inflated claims made headlines around the world. “Got ME? Just get out and exercise, say scientists,” wrote the Independent, using the acronym for the international name of the disease, myalgic encephalomyelitis. (Federal agencies now call it ME/CFS.) The findings went on to influence treatment recommendations from the CDC, the Mayo Clinic, Kaiser, the British National Institute for Health and Care Excellence, and more.

But patients like me were immediately skeptical, because the results contradicted the fundamental experience of our illness: The hallmark of ME/CFS is that even mild exertion can increase all the other symptoms of the disease, including not just profound fatigue but also cognitive deficits, difficulties with blood pressure regulation, unrestorative sleep, and neurological and immune dysfunction, among others.

Soon after I was diagnosed in 2006, I figured out that I had to rest the moment I thought, “I’m a little tired.” Otherwise, I would likely be semi-paralyzed and barely able to walk the next day.

The researchers argued that patients like me, who felt sicker after exercise, simply hadn’t built their activity up carefully enough. Start low, build slowly but steadily, and get professional guidance, they advised. But I’d seen how swimming for five minutes could sometimes leave me bedbound, even if I’d swum for 10 minutes without difficulty the day before. Instead of trying to continually increase my exercise, I’d learned to focus on staying within my ever-changing limits — an approach the researchers said was all wrong.

A disease ‘all in my head’?

The psychotherapy claim also made me skeptical. Talking with my therapist had helped keep me from losing my mind, but it hadn’t kept me from losing my health. Furthermore, the researchers weren’t recommending ordinary psychotherapy — they were recommending a form of cognitive behavior therapy that challenges patients’ beliefs that they have a physiological illness limiting their ability to exercise. Instead, the therapist advises, patients need only to become more active and ignore their symptoms to fully recover.

In other words, while the illness might have been triggered by a virus or other physiological stressor, the problem was pretty much all in our heads.

By contrast, in the American research community, no serious researchers were expressing doubts about the organic basis for the illness. Immunologists found clear patterns in the immune system, and exercise physiologists were seeing highly unusual physiological changes in ME/CFS patients after exercise.

I knew that the right forms of psychotherapy and careful exercise could help patients cope, and I would have been thrilled if they could have cured me. The problem was that, so far as I could tell, it just wasn’t true.

A deeply flawed study

Still, I’m a science writer. I respect and value science. So the PACE trial left me befuddled: It seemed like a great study — big, controlled, peer-reviewed — but I couldn’t reconcile the results with my own experience.

So I and many other patients dug into the science. And almost immediately we saw enormous problems.

Before the trial of 641 patients began, the researchers had announced their standards for success — that is, what “improvement” and “recovery” meant in statistically measurable terms. To be considered recovered, participants had to meet established thresholds on self-assessments of fatigue and physical function, and they had to say they felt much better overall.

But after the unblinded trial started, the researchers weakened all these standards, by a lot. Their revised definition of “recovery” was so loose that patients could get worse over the course of the trial on both fatigue and physical function and still be considered “recovered.” The threshold for physical function was so low that an average 80-year-old would exceed it.

In addition, the only evidence the researchers had that patients felt better was that patients said so. They found no significant improvement on any of their objective measures, such as how many patients got back to work, how many got off welfare, or their level of fitness.

But the subjective reports from patients seemed suspect to me. I imagined myself as a participant: I come in and I’m asked to rate my symptoms. Then, I’m repeatedly told over a year of treatment that I need to pay less attention to my symptoms. Then I’m asked to rate my symptoms again. Mightn’t I say they’re a bit better — even if I still feel terrible — in order to do what I’m told, please my therapist, and convince myself I haven’t wasted a year’s effort?

Many patients worked to bring these flaws to light: They wrote blogs; they contacted the press; they successfully submitted carefully argued letters and commentaries to leading medical journals. They even published papers in peer-reviewed scientific journals.

They also filed Freedom of Information Act requests to gain access to the trial data from Queen Mary University of London, the university where the lead researcher worked. The university denied most of these, some on the grounds that they were “vexatious.”

Critics painted as unhinged

The study’s defenders painted critics as unhinged crusaders who were impeding progress for the estimated 30 million ME/CFS patients around the world. For example, Richard Horton, the editor of the Lancet, described the trial’s critics as “a fairly small, but highly organised, very vocal and very damaging group of individuals who have, I would say, actually hijacked this agenda and distorted the debate so that it actually harms the overwhelming majority of patients.”

Press reports also alleged that ME/CFS researchers had received death threats, and they lumped the PACE critics in with the purported crazies.

While grieving for my fellow patients, I seethed at both the scientists and the journalists who refused to examine the trial closely. I could only hope that, eventually, PACE would drown under a slowly rising tide of good science, even if the scientific community never recognized its enormous problems.

But with the National Institutes of Health only funding $5 million a year of research into chronic fatigue syndrome, it seemed like that could take a very long time.

Then last October, David Tuller, a lecturer in public health and journalism at the University of California, Berkeley, wrote in Virology Blog a devastating expose of the scientific flaws of the trial. Tuller described all the problems I had seen, along with several more. The project was a remarkable act of public service: He isn’t a patient, yet he spent a year investigating the trial without institutional support, legal backing, or remuneration.

And, at last, the criticisms gained traction.

Racaniello and 41 other scientists and clinicians published an open letter to the Lancet calling for an independent investigation into the trial and saying “such flaws have no place in published research.” Rebecca Goldin, the director of Stats.org, an organization that works to improve the use of statistics in journalism, eviscerated the trial’s design in a 7,000-word critique.

In the meantime, a Freedom of Information Act request from Australian patient Alem Matthees was making its way through the legal system.

Matthees had asked for the anonymized data necessary to analyze the study using its original standards for success, but Queen Mary University of London had refused the request, arguing that malicious patients would break the anonymization and publish the participants’ names to discredit the trial. It again cited the death threats.

The court rejected these claims a month ago, calling them “wild speculations” and pointing out that the researchers themselves acknowledged in court that neither they nor PACE participants had received death threats.

Startling results from a re-analysis

Just before releasing the data,Queen Mary University of London did its own re-analysis on the question of how many patients got better, at least a little bit. Their data showed that using the study’s original standards, only 20 percent of patients improved with cognitive behavior therapy or exercise in addition to medical care, not 60 percent as claimed in the Lancet.

And even the 20 percent figure might be misleading, because the re-analysis also found that 10 percent of participants improved after receiving only standard medical care. That suggests that 10 percent in each of the treatment groups would likely have improved even without the exercise or therapy, leaving only 10 percent who were significantly helped by those interventions.

As for the claim that 22 percent of patients who received either treatment made an actual recovery? That went up in smoke when Matthees analyzed the raw data with the help of his colleagues and statisticians Philip Stark of the University of California, Berkeley, and Bruce Levin of Columbia University.

Newsletters

Sign up to our Weekend Reads newsletter

Please enter a valid email address.

Their analysis showed that had the researchers stuck to their original standards, only 4.4 percent of the exercise patients and 6.8 percent of the cognitive behavior therapy patients would have qualified as having recovered, along with 3.1 percent of patients in a trial arm that received neither therapy.

Importantly, there was no statistically significant difference between these recovery rates.

The PACE researchers, the editor of the Lancet, and the editors of Psychological Medicine (which published the follow-up study on recovery) all declined to comment for this article.

Simon Wessely, president of the UK Royal College of Psychiatrists, defended the trial in an email exchange with me. He argued that some patients did improve with the help of cognitive behavior therapy or exercise, and noted that the improvement data, unlike the recovery data, was statistically significant. “The message remains unchanged,” he wrote, calling both treatments “modestly effective.”

Wessely declined to comment on the lack of recovery. He summarized his overall reaction to the new analysis this way: “OK folks, nothing to see here, move along please.”

‘A classic bad study’

But it doesn’t appear that outside researchers are ready to “move along.”

After reviewing the new analysis, Jonathan Edwards, a professor emeritus of medicine at University College London said he was unconvinced that these small subjective improvements indicated the patients genuinely felt better. “They’ve set this trial up to give the strongest possible chance of there being a placebo effect that you can imagine,” he said.

“This is a classic bad study,” said Ron Davis, director of the Stanford Genome Technology Center and director of the Science Advisory Board of the End ME/CFS Project. He emphasized an additional problem: The study used such a broad definition of the disease that it likely included many patients who didn’t truly have ME/CFS at all.

“The study needs to be retracted,” Davis said. “I would like to use it as a teaching tool, to have medical students read it and ask them, ‘How many things can you find wrong with this study?’”

Retractions are rare, however, and erasing the impact of this flawed research will take much work for years to come.

After a sustained effort by ME/CFS advocates, the federal Agency for Healthcare Research and Quality, just changed its recommendation to read that there is insufficient evidence to justify cognitive behavior therapy or graded exercise. But many more public health agencies continue to point patients toward them.

And efforts to propagate this approach continue: A trial of graded exercise in children with ME/CFS has recently begun, and patients are protesting it.

Watching the PACE trial saga has left me both more wary of science and more in love with it. Its misuse has inflicted damage on millions of ME/CFS patients around the world, by promoting ineffectual and possibly harmful treatments and by feeding the idea that the illness is largely psychological. At the same time, science has been the essential tool to repair the problem.

But we shouldn’t take solace in the comforting notion that science is self-correcting. Many people, including many very sick people, had to invest immense effort and withstand vitriol to use science to correct these mistakes. And even that might not have been enough without Tuller’s rather heroic investigation. We do not currently have a sustainable, reliable method of overturning flawed research.

And rectifying PACE will take more than exposing its flaws. The lingering doubt it has cast on the illness will only be fully dispersed when we’ve finally figured out what’s really going on with the disease.

For that, we need to invest in some serious, good science. The kind I continue to love.

Julie Rehmeyer is a math and science writer. Her memoir “Through the Shadowlands,” describing the science and politics of chronic fatigue syndrome and other poorly understood illnesses, will be published by Rodale in May.

Leave a Comment

Please enter your name.
Please enter a comment.

  • I hate to post the top submission of the past year to the cat comment, but it was the first thing I thought of and went to grab

  • I love this article. I have shared it on 2 different sites with offers to translate if anyone would like. I am now following on fb (I have no tweet) and I will be purchasing the memoirs.
    I have been thinking this and saying this for years but no one believed me.
    Thank you so much for the details of what happened with the study definitions. I can’t research that knowledge.
    Blessedb.

  • Cycling Valtex completely eliminated symptoms of my CFS.

    I take 1g of valtrex twice a day(every 12 hours) for 4 consecutive days, once a month. I am a male and weigh about 180lb. ATTN, recovery mechnism is delayed, for the first 6-8 weeks I saw no improvement, and then I had full recovery. Also, I tried 500mg – it does not work for me.

    Here is my story in greater detail:

    My condition started with a flue and then continued with all the standard horrific symptoms. After two years of research and a couple of molecular cell biology courses, I narrowed my hypothesis down to Epstein-barr virus and HHV1 – high antibody titters kept on showing up in my bloodwork. Based on the information I could find, it seemed that EBV is suppressed by Valtres only at high concentrations while HHV1 is suppressed at both high and low concentrations. So, I started taking Valtrex – 1g ever 8 hours. After a couple of weeks i clearly felt that my body was suffering side effects from Valtrex. My stool turned white and I started loosing weight. I stopped after a month. And then, about 6-8(approximate) weeks since I first started (and so 2-4 weeks after I stopped taking Valtrex) I had a full recovery. Energy, comfort of being in my own body and sleep all returned. That was so odd! Recovery lasted about 6 weeks, and then I relapsed back into CFS. So, I started experimenting, and
    to keep the long story short, I discovered that the minimum dose was 7 1g pills of valtrex taken consecutively at 12 hour intervals ( I take 8 just in case). Recovery is delayed by 6-8 weeks, and then lasts another 6 weeks. So, to keep myself from relapsing I started taking it every month as described. And I have been feeling well since.

    I know Valtrex does not work for everybody, and I know that it works for some. I never tried Valcyte, I don’t know if cycling that in a similar way helps with other types of hhv

    • Review the book by Kent Heckenlively and Judy Mikovits “Plague – One scientists search for the truth…”. Amazon around $15 Aust. This might prove interesting with your training. Book reads like a novel but is based on her experiences as a researcher into CFS in America.
      Discusses XMRV retrovirus infection and Judy Mikovits experiences as she tried to pursue that line of scientific enquiry, and the challenges she faces from the research community and others. Early Mikovits research and others suggested that XMRX retrovirus was in many CFS patients and even a few percent of the general population. She concluded that XMRV or something similar lies dormant in human tissue and may be activated by stress to the immune system such as a virus or vaccine etc may activate the virus. This line of enquire has now been officially rejected by the scientific community but the book suggests why that might be. As I said interesting reading.

  • Which, if any, journal was responsible for publishing the flawed autism/vaccine study. That study, while long exposed as problematic, still haunts public policy and causes people to make poor choices. Why do we choose to believe false information even after it has long been proven wrong?

    • People who choose to believe in false things are not easily persuaded otherwise. The Germ Theory of Disease dates from the 1870’s. Two decades later, President McKinley died of septic shock, after a surgeon who did not believe the Germ Theory of Disease, operated on him with dirt under his fingernails, and reached inside the wound with those dirty nails. McKinley survived the surgery (a bullet fired by Leon Czolgosz, who was trying to win the affections of Syndicalist leader Emma Goldman, had to be removed) and seemed to be rallying. He got infected and died a few days later.

      Germ doubters like the surgeon who killed President McKinley, had various alternative theories about the causes of what we now know to be infections. The most common was that people can cure an infection by sheer force of will. Accepting the fact that washing the surgeon and the patient before surgery, to remove pathogens, prevented infection, means throwing away any ideas about how the patient’s force of will prevented infection. This exercise in dumb thinking, sustains the false belief and obstructs the recognition of facts.

      Pain patients today face the revival of that same dumb thinking. There are many people, particularly amongst the insurance lobby, who believe that chronic pain is a kind of hobby in which patients engage. If we can be persuaded to stop noticing the pain, we will feel better, goes the theory. And almost a third of the theory is based in fact…people with severe pain are given opioids, which actually enable the brain’s pain sensing center, the periaqueductal grey matter, to ignore most of the pain. The pain-as-hobby crowd are convinced that we patients could learn to do that, without the use of drugs, and are further convinced, that by inflicting a lot more pain and fear upon us, they can force us to ignore the pain. That remaining 2/3 of the belief system is hogwash. But getting the pain-as-hobby crowd to understand why their stupid idea doesn’t work, has proven very challenging.

      It’s comforting to believe, when witnessing death, that the dead person had a weakness that we don’t have. This belief makes us think that we are immune to dying. So, people continue believing dumb things that are false, but enable them to believe that they won’t die.

  • Julie,

    I read your article about ME/CFS on statnews.com with great interest and can in many ways I deeply relate to the experience of ME/CFS sufferers.

    I do not suffer from ME/CFS. However I relate to your story through my experience with DVS (Degenerative Vitreous Syndrome). It is a term sometimes used by the community for the severe form of eye floaters.

    I have suffered with severe eye floaters for almost a year now, and have been dismissed by doctors as having a psychosomatic problem.

    However, I have talked to many other sufferers who are also suffering from DVS. I do not believe for one moment that my problem is psychosomatic. Our experience with the condition simply does not agree with what the doctors say.

    With DVS, there is great difficulty with driving and my work in IT due to the severe visual disturbances. I have had to stop working due to it. There’s virtually no spot in my vision that is free of clouds of swirling debris. Simply going outdoors during the day causes great distress, and avoid going out into bright light as it causes intense discomfort. I can’t do something as simple as enjoy the scenery. It is no less than an assault on the senses.

    I have endured almost one year with these floaters, and have been to psychologists and psychiatrists. After spending a lot of money and time on CBT, I feel no better (and sometimes worse after sessions). I just feel that my psychologist and doctors are completely out of touch.

    Being diagnosed – I strongly believe incorrectly – with a psychosomatic disorder has caused me much more harm than good. I now feel very alienated by the medical profession and it has additional distress in addition to my original problems in day-to-day living with these floaters. This is worrisome, as it puts sufferers directs sufferers down the wrong track and thus has the potential to cause more harm.

    I am writing in the hope that your online magazine can in some way, small or big, shed some light on the struggle that many are having with DVS.

    While I’m grateful I still am able to see and understand that there are worse diseases, our community is nevertheless desperate for awareness for this often debilitating and invisible eye condition.

    This site explains what DVS is: http://www.oneclearvision.org
    This online petition explains more about DVS: https://www.change.org/p/research-and-awareness-for-severe-eye-floaters
    This forum contains discussions amongst DVS sufferers: http://floatertalk.yuku.com/

    • Is there not a medical treatment for this? My father had what was described as essentially a “vacuum” applied to one of his eyes to remove debris/floaters.

    • A vitrectomy is a procedure that is used to remove the vitreous gel from the eye. It has a very high success rate of significantly improving or resolving entirely DVS symptoms. However, it is a procedure that carries significant risk to sight. If – and a big if – you can find a GP willing to consider the possibility that it isn’t a psychosomatic disorder, then you still need to find a surgeon willing to perform the surgery. Depending on where you live, you may or may not be eligible for full or partial insurance assistance, because it is generally seen as a ‘benign’ condition, despite the debiliting effects of the condition. Recent research papers have shown some of the deleterious effects of symptomatic opacities in the eye.

      In any case, while psychological approaches should be considered, my experience tells me that more rigorous criteria needs to be met before labelling anyone as having a psychosomatic disorder and starting treatment that may inadvertently worsen health.

      There is not very much literature in the field on DVS, but Wagle et al shed some light on the quality of life effects of symptomatic vitreous opacities on patients.

    • Hi Will,

      I’m just curious whether your father had other complications of the eye that led to the vitrectomy? I know that vitrectomy is a routine procedure done for emergencies such as retinal breaks or bleeding etc.

      Regards

  • My mother and I became ill with cfs and irritable bowel symptoms in Dec. 2010 within hours of each other. We were diagnosed by colonoscopy and biopsies in Feb. 2011 with an enterovirus infection. We have had some improvement in bowel symptoms since then but other symptoms are about the same. The Enterovirus Foundation and Enteroviruses.com have been very helpful in understanding our disease.

    • I was first diagnosed in 1990 as having chronic fatigue syndrome and now the diagnosis is fibromyalgia and every doctor I see wants me to take Lyrica is there politics and money moving this medication, which made me dizzy, wobbly, and sicker?

  • Thank You . Glad to hear that eventually good science will be triumphant. I an hoping that the Lancet will see the writing on the wall and retract the pace study . I was originally diagnosed with fibromyalgia then Cfs. The pace study definitely effected my treatment. For the first 3 years I dutifully went to doctors and specialists. As a pharmacist I was taught evidence based medicine but now older and wiser it is dismaying to me the extent of influence ” politics and money” can have and unfortunately science and healthcare is not immune. I never take anything at face value ( even respected medical journals ( The Lancet) can be influenced), need to be a detective as well as a scientist.

    I have made much improvement mostly by researching and becoming my own doctor. My family doctor supports me. Got rid of fibromyalgia with Celebrex and Valtrex. Got rid of ” Cfs with mold avoidance and cholystyramine. I had an extremely bad experience and I have a supportive spouse and money in the bank.

    I can only imagine the suffering this Pace study has caused. Thank you and David Tuller for exposing what patients have been saying for 20 years.

    • As told in Surviving Mold by Dr. Ritchie Shoemaker, it didn’t bother me that Cheney and Peterson weren’t interested in the Sick Building component of the 1985 “Tahoe Mystery Illness”.

      I thought when the new syndrome became known, “Other researchers will come, eager for clues”

      History has proven this not to be the case.

      Their only interest was to clear them out of the way, to make CFS safe for their own theory, whatever it may be.

      My plan was.. that IF any researcher ever did want to find out the whole story, I could put them in touch with the remediators who cleaned up the schools.

      And then, by combining documented reports, medical records, the new found evidence of what mold is capable of, we could move toward solving the mystery, as is the avowed purpose of writing an abstract or creating a new syndrome.

      That plan has fallen through.

      I called up the remediators involved, to ask if they would step up and explain the sick buildings, and was told they can not do this.

      “For reasons of confidentiality”

      Even if I bring CFS researchers and these remediators together, they not only have no desire to solve CFS, for reasons of their own vested interests, but are prevented by contractual agreements.

      Since I am the only one with no attachments pursuing this, and under no obligation to remain silent, it appears that I am the only chance left to get these facts.

      I am the last witness who is willing to speak.

    • I have no scientific evidence for this hypothesis … but I would guess that cholestyramine, which is a bile acid sequestering agent, binds bile acids in the gut, which in turn, alters the microbiome. Just a guess, though.

Sign up for our Morning Rounds newsletter

Your daily dose of news in health and medicine