If your doctor diagnoses you with chronic fatigue syndrome, you’ll probably get two pieces of advice: Go to a psychotherapist and get some exercise. Your doctor might tell you that either of those treatments will give you a 60 percent chance of getting better and a 20 percent chance of recovering outright. After all, that’s what researchers concluded in a 2011 study published in the prestigious medical journal the Lancet, along with later analyses.
Problem is, the study was bad science.
And we’re now finding out exactly how bad.
Under court order, the study’s authors for the first time released their raw data earlier this month. Patients and independent scientists collaborated to analyze it and posted their findings Wednesday on Virology Blog, a site hosted by Columbia microbiology professor Vincent Racaniello.
The analysis shows that if you’re already getting standard medical care, your chances of being helped by the treatments are, at best, 10 percent. And your chances of recovery? Nearly nil.
The new findings are the result of a five-year battle that chronic fatigue syndrome patients — me among them — have waged to review the actual data underlying that $8 million study. It was a battle that, until a year ago, seemed nearly hopeless.
When the Lancet study, nicknamed the PACE trial, first came out, its inflated claims made headlines around the world. “Got ME? Just get out and exercise, say scientists,” wrote the Independent, using the acronym for the international name of the disease, myalgic encephalomyelitis. (Federal agencies now call it ME/CFS.) The findings went on to influence treatment recommendations from the CDC, the Mayo Clinic, Kaiser, the British National Institute for Health and Care Excellence, and more.
But patients like me were immediately skeptical, because the results contradicted the fundamental experience of our illness: The hallmark of ME/CFS is that even mild exertion can increase all the other symptoms of the disease, including not just profound fatigue but also cognitive deficits, difficulties with blood pressure regulation, unrestorative sleep, and neurological and immune dysfunction, among others.
Soon after I was diagnosed in 2006, I figured out that I had to rest the moment I thought, “I’m a little tired.” Otherwise, I would likely be semi-paralyzed and barely able to walk the next day.
The researchers argued that patients like me, who felt sicker after exercise, simply hadn’t built their activity up carefully enough. Start low, build slowly but steadily, and get professional guidance, they advised. But I’d seen how swimming for five minutes could sometimes leave me bedbound, even if I’d swum for 10 minutes without difficulty the day before. Instead of trying to continually increase my exercise, I’d learned to focus on staying within my ever-changing limits — an approach the researchers said was all wrong.
A disease ‘all in my head’?
The psychotherapy claim also made me skeptical. Talking with my therapist had helped keep me from losing my mind, but it hadn’t kept me from losing my health. Furthermore, the researchers weren’t recommending ordinary psychotherapy — they were recommending a form of cognitive behavior therapy that challenges patients’ beliefs that they have a physiological illness limiting their ability to exercise. Instead, the therapist advises, patients need only to become more active and ignore their symptoms to fully recover.
In other words, while the illness might have been triggered by a virus or other physiological stressor, the problem was pretty much all in our heads.
By contrast, in the American research community, no serious researchers were expressing doubts about the organic basis for the illness. Immunologists found clear patterns in the immune system, and exercise physiologists were seeing highly unusual physiological changes in ME/CFS patients after exercise.
I knew that the right forms of psychotherapy and careful exercise could help patients cope, and I would have been thrilled if they could have cured me. The problem was that, so far as I could tell, it just wasn’t true.
A deeply flawed study
Still, I’m a science writer. I respect and value science. So the PACE trial left me befuddled: It seemed like a great study — big, controlled, peer-reviewed — but I couldn’t reconcile the results with my own experience.
So I and many other patients dug into the science. And almost immediately we saw enormous problems.
Before the trial of 641 patients began, the researchers had announced their standards for success — that is, what “improvement” and “recovery” meant in statistically measurable terms. To be considered recovered, participants had to meet established thresholds on self-assessments of fatigue and physical function, and they had to say they felt much better overall.
But after the unblinded trial started, the researchers weakened all these standards, by a lot. Their revised definition of “recovery” was so loose that patients could get worse over the course of the trial on both fatigue and physical function and still be considered “recovered.” The threshold for physical function was so low that an average 80-year-old would exceed it.
In addition, the only evidence the researchers had that patients felt better was that patients said so. They found no significant improvement on any of their objective measures, such as how many patients got back to work, how many got off welfare, or their level of fitness.
But the subjective reports from patients seemed suspect to me. I imagined myself as a participant: I come in and I’m asked to rate my symptoms. Then, I’m repeatedly told over a year of treatment that I need to pay less attention to my symptoms. Then I’m asked to rate my symptoms again. Mightn’t I say they’re a bit better — even if I still feel terrible — in order to do what I’m told, please my therapist, and convince myself I haven’t wasted a year’s effort?
Many patients worked to bring these flaws to light: They wrote blogs; they contacted the press; they successfully submitted carefully argued letters and commentaries to leading medical journals. They even published papers in peer-reviewed scientific journals.
They also filed Freedom of Information Act requests to gain access to the trial data from Queen Mary University of London, the university where the lead researcher worked. The university denied most of these, some on the grounds that they were “vexatious.”
Critics painted as unhinged
The study’s defenders painted critics as unhinged crusaders who were impeding progress for the estimated 30 million ME/CFS patients around the world. For example, Richard Horton, the editor of the Lancet, described the trial’s critics as “a fairly small, but highly organised, very vocal and very damaging group of individuals who have, I would say, actually hijacked this agenda and distorted the debate so that it actually harms the overwhelming majority of patients.”
Press reports also alleged that ME/CFS researchers had received death threats, and they lumped the PACE critics in with the purported crazies.
While grieving for my fellow patients, I seethed at both the scientists and the journalists who refused to examine the trial closely. I could only hope that, eventually, PACE would drown under a slowly rising tide of good science, even if the scientific community never recognized its enormous problems.
But with the National Institutes of Health only funding $5 million a year of research into chronic fatigue syndrome, it seemed like that could take a very long time.
Then last October, David Tuller, a lecturer in public health and journalism at the University of California, Berkeley, wrote in Virology Blog a devastating expose of the scientific flaws of the trial. Tuller described all the problems I had seen, along with several more. The project was a remarkable act of public service: He isn’t a patient, yet he spent a year investigating the trial without institutional support, legal backing, or remuneration.
And, at last, the criticisms gained traction.
Racaniello and 41 other scientists and clinicians published an open letter to the Lancet calling for an independent investigation into the trial and saying “such flaws have no place in published research.” Rebecca Goldin, the director of Stats.org, an organization that works to improve the use of statistics in journalism, eviscerated the trial’s design in a 7,000-word critique.
In the meantime, a Freedom of Information Act request from Australian patient Alem Matthees was making its way through the legal system.
Matthees had asked for the anonymized data necessary to analyze the study using its original standards for success, but Queen Mary University of London had refused the request, arguing that malicious patients would break the anonymization and publish the participants’ names to discredit the trial. It again cited the death threats.
The court rejected these claims a month ago, calling them “wild speculations” and pointing out that the researchers themselves acknowledged in court that neither they nor PACE participants had received death threats.
Startling results from a re-analysis
Just before releasing the data,Queen Mary University of London did its own re-analysis on the question of how many patients got better, at least a little bit. Their data showed that using the study’s original standards, only 20 percent of patients improved with cognitive behavior therapy or exercise in addition to medical care, not 60 percent as claimed in the Lancet.
And even the 20 percent figure might be misleading, because the re-analysis also found that 10 percent of participants improved after receiving only standard medical care. That suggests that 10 percent in each of the treatment groups would likely have improved even without the exercise or therapy, leaving only 10 percent who were significantly helped by those interventions.
As for the claim that 22 percent of patients who received either treatment made an actual recovery? That went up in smoke when Matthees analyzed the raw data with the help of his colleagues and statisticians Philip Stark of the University of California, Berkeley, and Bruce Levin of Columbia University.
Their analysis showed that had the researchers stuck to their original standards, only 4.4 percent of the exercise patients and 6.8 percent of the cognitive behavior therapy patients would have qualified as having recovered, along with 3.1 percent of patients in a trial arm that received neither therapy.
Importantly, there was no statistically significant difference between these recovery rates.
The PACE researchers, the editor of the Lancet, and the editors of Psychological Medicine (which published the follow-up study on recovery) all declined to comment for this article.
Simon Wessely, president of the UK Royal College of Psychiatrists, defended the trial in an email exchange with me. He argued that some patients did improve with the help of cognitive behavior therapy or exercise, and noted that the improvement data, unlike the recovery data, was statistically significant. “The message remains unchanged,” he wrote, calling both treatments “modestly effective.”
Wessely declined to comment on the lack of recovery. He summarized his overall reaction to the new analysis this way: “OK folks, nothing to see here, move along please.”
‘A classic bad study’
But it doesn’t appear that outside researchers are ready to “move along.”
After reviewing the new analysis, Jonathan Edwards, a professor emeritus of medicine at University College London said he was unconvinced that these small subjective improvements indicated the patients genuinely felt better. “They’ve set this trial up to give the strongest possible chance of there being a placebo effect that you can imagine,” he said.
“This is a classic bad study,” said Ron Davis, director of the Stanford Genome Technology Center and director of the Science Advisory Board of the End ME/CFS Project. He emphasized an additional problem: The study used such a broad definition of the disease that it likely included many patients who didn’t truly have ME/CFS at all.
“The study needs to be retracted,” Davis said. “I would like to use it as a teaching tool, to have medical students read it and ask them, ‘How many things can you find wrong with this study?’”
Retractions are rare, however, and erasing the impact of this flawed research will take much work for years to come.
After a sustained effort by ME/CFS advocates, the federal Agency for Healthcare Research and Quality, just changed its recommendation to read that there is insufficient evidence to justify cognitive behavior therapy or graded exercise. But many more public health agencies continue to point patients toward them.
And efforts to propagate this approach continue: A trial of graded exercise in children with ME/CFS has recently begun, and patients are protesting it.
Watching the PACE trial saga has left me both more wary of science and more in love with it. Its misuse has inflicted damage on millions of ME/CFS patients around the world, by promoting ineffectual and possibly harmful treatments and by feeding the idea that the illness is largely psychological. At the same time, science has been the essential tool to repair the problem.
But we shouldn’t take solace in the comforting notion that science is self-correcting. Many people, including many very sick people, had to invest immense effort and withstand vitriol to use science to correct these mistakes. And even that might not have been enough without Tuller’s rather heroic investigation. We do not currently have a sustainable, reliable method of overturning flawed research.
And rectifying PACE will take more than exposing its flaws. The lingering doubt it has cast on the illness will only be fully dispersed when we’ve finally figured out what’s really going on with the disease.
For that, we need to invest in some serious, good science. The kind I continue to love.
Julie Rehmeyer is a math and science writer. Her memoir “Through the Shadowlands,” describing the science and politics of chronic fatigue syndrome and other poorly understood illnesses, will be published by Rodale in May.
“And indeed I happen to think the changes were right – the criteria they settled on gave results much more congruent with previous studies and indeed routine outcome measure studies of which there are many.”
Simon, are you trying to justify protocol changes simply because they made the results look as the authors wanted them to, ie in line with previous much smaller studies? That’s an absolutely astonishing admission.
Also, your claim about recovery and the changes reflecting a more meaningful definition of recovery is equally staggering. Not only was the reduction to 60 on the physical function scale calculated based on a basic mistake related to the physical function of the working age and adult age population, you are also essentially claiming that patients who entered the trial classed as having significant, life changing disability could get worse and be classed as ‘recovered’ or, in the words of Trudie Chalder, ‘back to normal’. That you make a point that is so absurd genuinely defies belief.
All objective measures, including welfare, employment and fitness were either sidelined or abandoned and the six minute walking test showed no benefit for Cbt and little for GET.
What’s more, the authors’ own reanalysis, rushed out just before the data had to be legally released, showed that the protocol changes had inflated the ‘improver’ figures by 200%. We now discover that the ‘recovery’ stats were boosted by as much as 300%.
Just saying that you are “sorry to spoil the party” with “cold, hard, facts” does not make your defence of the indefensible any more absurd.
You say that CBT and GET are the best treatments studied. 1) pacing undertaken by patients does not correspond to APT. 2) harms have never been adequately reported. 3) Funding for biomedical research into the condition has been pitiful to this point. Indeed, only now are large scale trials making intriguing findings. If two treatments are the only ones studied, in trials designed and conducted by researchers of a like mind, using deliberately broad criteria (and, like PACE, including patients with co-morbid psychiatric conditions which respond to those two treatments) how could Cbt and GET not be the ‘best treatments we have’?
Despite constructing a trial, using the broadest criteria for the condition possible to create heterogeneity, discarding objective outcomes, relying on only subjective ones (when CBT is deliberately designed to make the patient minimise their symptoms) and priming patients in mid-trial leaflets (which lauded CBT as a treatment for the illness) PACE still produced results so poor that placebo alone could explain them. To then suggest that protocol changes that inflated findings by 3-4 times are not significant is insulting to patients and to the many researchers and statisticians who are agog at what has happened here.
Your responses here insult the intelligence. That the UK research establishment remain silent whilst criticism rains down from abroad tells you all you need to know about this little island and it’s old boys network. The repeated message that it is a minority of militant patients trying to deny that they have a psychiatric illness was roundly rebuffed by the Information Tribunal, whose decision is a must read. Many CFS/ME patients have co-morbid psychiatric conditions about which they feel no shame. Frankly, they just want treatments that work. They want to be better and return to their old lives. If that happened to be a Prozac a day or an antipsychotic they would care not a jot. The reality is that lives have been destroyed by graded exercise programmes from which patients have never recovered. If these outcomes resulted from pharmaceuticals those drugs would be discontinued. Because this anecdotal evidence comes from patients with a contested illness their experiences are disregarded.
This is not about anything but finding treatments that genuinely work. PACE purports to do that, and it’s authors and their supporters continue to repeat that mantra, but the trial itself does not stand up to scientific rigour. That is not just the opinion of laymen patients, but experienced academics, medics and statisticians, scores of whom wrote to the Information Tribunal to implore that the PACE trial data be released to allow for reanalysis. The genie is out of the box and the above, I’m afraid, is a very intelligent, erudite man making ridiculously indefensible points in the hope that his reputation and position will encourage medics and academics alike to dismiss the criticisms without ever scrutinising the trial themselves.
Actually, the double-blind study of the effects of Ampligen (an immune modulator) on patients with CFS (Fukuda 1994) showed statistically significant improvement of symptoms and increase in functioning, without any significant toxicities. So there IS another study out there about a treatment with better results than the PACE study – and it rested on biomedical rather than psychiatric assumptions about the disease.
More recently, the company (Hemispherx) has published a study showing that natural killer cell function is disrupted in ME/CFS. Using patients who do know they are on the drug (but were significantly sicker than those in the double-blind), they found that Ampligen can improve NK cell function.
So not only was the “definitive” study of CBT/GET – the PACE trial – a failure, but the study that was able to show statistically significant, if small, improvement rested on a biomedical approach to ME/CFS. We need many more studies – but I believe it is clear that the biopsychosocial psychiatric approach that gave us CBT/GET is at a dead end, and new funds should be directed at biomedical explanations for this disease.
Diagnosed in 2002. Graded activity approach helped me without a doubt to learn to know and listen to my own body. Acceptance of my weakness and to say no and mean no without guilt were also two keys to my recovery. Days now 100% good and enjoying life at a more slower, relaxed and happy pace. Others don’t understand, unless you have been through it – but that’s the norm with any trauma experience. For me it was a trauma experience and coming to a place of peace is a good place to be now at. I got a hep B vaccine a month prior to falling ill and I always wonder did that trigger the CFS?
Thoughtful, clearly presented and balanced argument. Very helpful.
No need to apologise, a bit of glibness never spoilt any party that I know of and such condescension is worth the ‘cold facts’ you pour upon us. And on the subject of ‘cold facts’, there were plenty provided by the UK and worldwide media as informed by the PACE authors at the ScMC press conference:
https://peterkempblog.wordpress.com/2016/09/23/the-science-media-centre-and-the-pace-trial-a-deceptive-partnership/
The link provides just a tiny selection which repeat the same misleading messages not supported by the data. Sharpe himself stated that the NNT was ‘about 7’ which cannot correlate with 60% or even 30%.
That’s what I call a party-pooper.
Simon Wessely, As you say, you played a part in the early development of CBT for this, or these, illnesses. I think some of the papers by you and your colleagues display conceptual flaws. I’m a logician and philosopher of science; my observations are most likely sound. With luck I will write on this. I’ll send you a copy.
@JG Collins: Take your head out of your arse. The planet is warming, last year had a significantly higher increase that previous years. Animal species from warmer regions of land and sea have been moving further North, year on year, for the last 50 years. Petrochemical industries have been falsifying data for decades.
http://climate.nasa.gov/evidence/
http://www.bloomberg.com/graphics/2015-whats-warming-the-world/
https://www.theguardian.com/environment/2014/oct/31/ipcc-report-six-graphs-that-show-how-were-changing-the-worlds-climate
Wow, what an amazing article.
In my opinion graded exercise therapy and psychological treatments are sorely misunderstood by practitioners and even those who study them. Graded exercise therapy can only be expected to work when you monitor ALL exertion. Psychological treatments can only be expected to work when they manage the physiological stress response which is so damaging to the nervous and immune systems.
I am a 31 year old graduate in the history, philosophy and social studies of science and I have had CFS/ME for 4 years. I read this article with enthusiastic interest and feel I must share my story.
As a typical UK sufferer I have had a difficult journey through the health system and believe my continued illness is a result of that difficulty. Despite currently being as ill as I’ve ever been I do believe that I have found a potential inroad to an effective recovery regime based on what I have learned about psychological and graded exercise therapy treatments.
I am the type of person who thinks about things precisely and when I first learned about graded exercise therapy it immediately seemed wrong to me for the simple reason of how exercise, or more correctly exertion, was measured. Graded exercise therapy as it was explained to me by the highly commendable Sussex CFS/ME Service requires choosing a task such as walking to the shops that you would like to build up to. You then begin a regime of incremental walks at a pace you can manage until you reach your goal. Fine in theory until you realise that the practice of this is that you should take one incremental walk a day on top of what you would usually do in a day.
This has been set with the best of intentions as it is on the surface the only practical way of measuring daily exercise. However, when you put a lense of precision on this you find it is woefully inaccurate. The revolution came to me when my friend suggested I buy a Fitbit activity tracker, essentially a digital pedometer.
A Fitbit is designed to be worn 24/7 and even monitors your sleep. In a matter of weeks my suspicions about graded exercise therapy were confirmed in the beautiful scientific expression of statistics and graphs, albeit very personal ones. Never mind a daily exercise session, the much talked about “boom bust cycle” could be so clearly seen happening through a GET routine because of the exertion you do in the rest of the day.
A simple idea of a day in the house as being no exercise does not work because you could have spent all day on the sofa or you could have been drifting from room to room actually walking the equivalent of to the shops and back. I live in a tiny flat and still get this effect.
I really believe that graded exercise therapy works but only if you monitor total daily exertion and now we have the technology to do this. In the year I have had my tracker I have developed a holistic routine based on monitoring my exertion and symptom levels and setting strict daily limits. Only when I did a course of graded exercise therapy 8 months ago using all my tracker data did everything finally fit into place.
I really want to explain more and to get my system out there into the CFS/ME community but my mental fatigue makes communicating so so hard. I have so much to say and so much more I want to learn but I have a brain full of fudge.
To conclude, going forward, what I want to see is trials doing ‘proper’ graded exertion therapy using digital activity trackers which would provide a wealth of truely objective data. The theory that we are rehabilitating our bodies to exertion isn’t ready to be thrown out but we need realistic measures. I’m aware I haven’t covered the psychological component which I believe is ultimately down to stress but I think I have written enough for now and it’s my rest time ?
Layla,
I hate to tell you this, but you are going to be a future bed-bound patient. No exercise treats ME/CFS in any way. And you can’t measure the exertion it takes to think which also causes post-exertion malaise. http://me-pedia.org/wiki/Post-exertional_malaise
These fit-bits are a nifty tech toy that everyone including fit-bit is trying to find a use for.
The disease has to be treated and vast improvement must be made before you can exercise or exert yourself.
And if you are under the UK Oxford criteria, you may not even have ME as the criteria sweeps in patients with Chronic Fatigue from a lot of other illnesses such as thyroid, adrenals, depression, drug therapy and on and on. The Oxford criteria is basically a chronic fatigue and mental health criteria and the whole problem starts right THERE! http://me-pedia.org/wiki/Oxford_criteria
Don’t believe for one second your Fit-Bit is going to monitor what is really going wrong with exertion. The only ability we have to measure all that is happening biologically and is in research.
Layla, this is activity management or pacing you describe, not graded exercise
Layla, try this link https://youtu.be/7BceGgEdMpA about the viability of anaerobic exercise rather than aerobic exercise.
@Sir Wessely, you want turn the table but the table is already broken. Your reply sounds to me as an Ad Hoc hypothesis wich is uasully common in pseudoscience like psychology. I am still wondering why White or Chalder don’t repul by themselves, do they really need your help?
One of th troubles with science is that it is often decades later that it’s truth or otherwise is confirmed. Then, more decades to show that the confirmation was wrong. Many scientists have been ridiculed, then the establishment realises that the person was right after all. That, in itself, is a deterrent for many, to speak up.
Taxpayers lost millions thanks to this blunder. Why weren’t researchers honest about their recovery findings?
These researchers are being paid by the disability insurers to be dishonest. The UK Parliament Group on Scientific Research into ME (the Gibson Enquiry) [1] called for a proper investigation into well-documented conflicts of interest TEN YEARS ago:
“”CFS/ME is defined as a psychosocial illness by the Department for Work and Pensions (DWP) and medical insurance companies. Therefore claimants are not entitled to the higher level of benefit payments. We recognise that if CFS/ME remains as one illness and/or both remain defined as psychosocial then it would be in the financial interest of both the DWP and the medical insurance companies.”
“[There are] numerous cases where advisors to the DWP have also had consultancy roles in medical insurance companies. Particularly the Company UNUMProvident. Given the vested interest private medical insurance companies have in ensuring CFS/ME remain classified as a psychosocial illness there is blatant conflict of interest here.” [2]
In the US, disability benefits for so-called “mental illness” is generally limited to two years, and that is why “CFS” is still considered to be “mostly psychological”, in spite of the huge body of research that clearly demonstrates serious biomedical anomalies.
[1] http://erythos.com/gibsonenquiry/Report.html
[2] https://en.wikipedia.org/wiki/Controversies_related_to_chronic_fatigue_syndrome#Political