BMJ should retract flawed research paper on chronic fatigue syndrome

Few journals have been more admirable than The BMJ (formerly the British Medical Journal) and some of its sister publications under the BMJ brand in highlighting issues of direct significance to health care consumers. So it is baffling — and troubling — when BMJ editors fail to take appropriate action to address unacceptable lapses in high-profile research they have published.

For years, the reading list for my journalism class on public health and medicine at the University of California, Berkeley, included groundbreaking articles in The BMJ on “disease-mongering” — how pharmaceutical companies have manipulated and misrepresented research data to expand existing diagnostic categories and create new ones. I have also appreciated BMJ’s campaign for open access to trial data and its forays into investigative journalism.

Much of this hard-hitting approach can be attributed to Dr. Fiona Godlee, the Cambridge University-educated physician who has led the organization since 2005. Godlee is both editorial director of BMJ, which publishes dozens of titles, and editor-in-chief of The BMJ, one of the world’s leading medical journals. A 2016 profile of Godlee in STAT called her a “crusading editor” who “aims to shake things up in science.” The BMJ, she told STAT, is “a campaigning journal.”

That’s why I am disappointed at how BMJ and Godlee have handled a seriously problematic paper in a field I know well — behavioral and psychological interventions for the illness (or cluster of illnesses) often called chronic fatigue syndrome (CFS) but also known as myalgic encephalomyelitis (ME), CFS/ME, and ME/CFS, among other names.

ME/CFS, as U.S. government agencies now refer to it, is a disabling disease marked by immunological, neurological, and other physiological dysfunctions. There are no currently identified diagnostic tests or approved drug treatments for it. The symptoms include a prolonged relapse following minimal amounts of exertion, cognitive impairments, and sleep disorders, among others.

For many years, the standard of care mainly involved psychotherapy and a graded exercise program. These treatments were based largely on the widely dismissed theory that patients’ ongoing symptoms were caused by deconditioning, a belief promulgated for decades by a group of U.K. psychiatrists and other health professionals. In the face of serious scientific questions about the research behind this rehabilitative approach, the Centers for Disease Control and Prevention dropped its recommendations for these treatments in 2017.

As a journalist and public health academic at Berkeley, I have written extensively about this issue, largely for a science site called Virology Blog, but also for the New York Times, Undark, and STAT, among others. My investigation has documented substantive methodological and ethical problems with a number of behavioral and psychological studies, some of them published in BMJ journals, suggesting that BMJ, like much of the British academic and medical establishment, might have a blind spot in this domain.

A striking example is a study of a commercial three-day self-help program called the Lightning Process as a treatment for pediatric CFS/ME (as the investigators called the illness). The Lightning Process is a goulash of neuro-linguistic programming, life coaching, osteopathy, positive affirmations, and other elements. It aims to train people to improve their health by altering their thought processes and refusing to allow their symptoms to control them.

Its founder, osteopath Phil Parker, previously taught aspiring healers how to diagnose people’s ailments through auras, tarot cards, energy fields, and the spinning of pendulums, as well as how to communicate with spiritual guides. Lightning Process practitioners have claimed to successfully treat people with multiple sclerosis, eating disorders, and other illnesses. U.K. advertising authorities have cited practitioners for making unsubstantiated medical claims.

The Lightning Process trial was not blinded and relied on self-reported outcomes — a design with enormous potential for bias, since participants’ knowledge of what treatment they received can influence their subjective responses to questions about their health. Nevertheless, based on their findings, the investigators declared that the Lightning Process, in combination with medical care, was an “effective” treatment for pediatric CFS/ME. Archives of Disease in Childhood, a well-regarded BMJ journal, published the results in September 2017.

Beyond questions about the Lightning Process as a treatment modality, the study itself violated core principles of scientific inquiry. The investigators recruited more than half of the participants before trial registration, swapped primary and secondary outcomes after gathering data from the early recruits, and then failed to disclose these critical details in the paper.

I noted these irregularities in a blog post in December 2017, three months after the study was published.

Recruiting participants before the trial was officially registered should have doomed the prospects for publication from the start. In 2005, the International Committee of Medical Journal Editors adopted a policy of rejecting trials in which any participants were recruited before formal registration in a recognized database.

Prospective trial registration requires investigators to provide in advance full details about study design: number of participants, the interventions, primary and secondary outcomes, and other details. Editors of the major journals agreed to this policy as a way of preventing selective outcome reporting — the tendency to extract and highlight the best results rather than sticking to a predesignated plan.

BMJ has been a leading voice for the effort to tighten prospective registration requirements. At a hearing of the U.K. Parliament in 2013, Godlee touted the publisher’s zero-tolerance policy. “As far as we are aware, in the last two years we have not published any trial that has not been prospectively registered,” she declared. In 2015, an editorial in The BMJ called the prospective registration policy “the single most valuable tool we have to ensure unbiased reporting of research studies.”

The Lightning Process study demonstrates what can happen when that “valuable tool” is abandoned. In the trial, the initial primary outcome was school attendance at six months; self-reported physical function was designated a secondary outcome. But after gathering data from children recruited before the trial was registered, the investigators made self-reported physical function the primary outcome and demoted school attendance to a secondary outcome. That bolstered their published findings because they were able to report positive results for self-reported physical function, their revised primary outcome. In contrast, school attendance at six months showed no improvement — a finding that was downplayed and essentially ignored. In other words, the mid-trial switch made it easier for the investigators to assert that the Lightning Process was effective.

By falsely presenting the study as a prospective trial with pre-designated assessment methods, the investigators misled the public, medical practitioners, and policymakers about their results. News accounts amplified this distorted message.

In January 2018, Archives of Disease in Childhood agreed to look into the matter. After a year and a half, the journal finally reissued the paper in July 2019 with an extensive correction along with a reiteration of the original results — that the Lightning Process is “effective.” An accompanying editor’s note sought to justify republishing the study with the same findings rather than retracting it.

The journal’s justifications for this decision cannot withstand scrutiny.

First, according to the editor’s note, “BMJ policy requires prospective registration of randomised trials but we do not consider a failure to enforce that policy grounds for retraction.” The note did not explain the reasoning behind this perplexing conclusion, which plainly conflicts with BMJ’s categorical declarations about the importance of prospective registration.

The editor’s note also tried to massage away the problem of the mid-trial outcome-swapping. The investigators had attributed the change to preferences expressed by early trial participants. According to the editor’s note, the journal sought “assurance from the authors that the change in primary outcome was not influenced by (positive) findings” during the first recruitment phase.

Asking those who might have been influenced by bias if they were, in fact, influenced by bias seems to willfully ignore the central point. Designing studies to minimize bias is important precisely because humans are unreliable assessors of whether our desires and beliefs have affected our decision-making. Such an “assurance” of lack of bias should not be taken at face value, especially when a study’s design itself promotes bias and the actions in question clearly improve a study’s reported results.

Neither the correction nor the editor’s note provided an adequate explanation for why the investigators did not share these salient facts in the published paper. Perhaps the journal decided that question was better left untouched. After all, the U.K.’s Medical Research Council, a major funding agency, defines research misconduct to include “misrepresentation of data, for example suppression of relevant findings and/or data, or knowingly, recklessly or by gross negligence, presenting a flawed interpretation of data.”

The Lightning Process investigators presented a “flawed interpretation of data” by not revealing what they actually did. Whether or not this “suppression” was done “knowingly,” it was certainly done “recklessly” and/or by “gross negligence.” Even if opinions differ on whether this case constitutes actual research misconduct, it is hard to understand how a team of experienced investigators could have omitted such relevant information and offered such an inaccurate account for publication.

Under the circumstances, continuing to grant the Lightning Process study the BMJ seal of approval can only be described as a form of “gross negligence” on the editorial side. In response to the correction, I appealed to Godlee in an open letter of concern in August 2019, signed by 55 scientists, academics, and other experts from Harvard, Columbia, Stanford, University College London, UC Berkeley, Queen Mary University of London, and elsewhere.

The letter called Archives of Disease in Childhood’s decision “scientifically and ethically indefensible” and “potentially harmful” to children, noting that the journal “has now re-published positive primary results generated by outcome-swapping in a study that did not meet a strict publication requirement [i.e. prospective registration] specifically intended to prevent outcome-swapping and other kinds of selective outcome reporting.” With no response from Godlee, I sent the letter again in November, this time endorsed by even more experts and over 60 patient and advocacy organizations from around the world.

Some of the signers also emailed Godlee directly to express dismay. “As you are aware, many other scientists of repute have signed on to the original letter to you,” wrote Vincent Racaniello, a professor of microbiology at Columbia and the host of Virology Blog. “I am sure we could have hundreds more. How many do you need before you are convinced that the work should be retracted? How much longer do you wish to risk harm to young patients?”

Dr. Ronald Tompkins, a professor of surgery at Harvard Medical School, informed Godlee that he was “appalled” that the study remains in the literature. And Brian Hughes, a psychology professor at the National University of Ireland, Galway, wrote Godlee that the journal’s apparent rewarding of questionable conduct “will strongly encourage other researchers to play fast and loose with pre-registration, in the knowledge that their findings can still be published in a BMJ journal.”

Despite such vigorous appeals, Godlee has now reaffirmed BMJ’s support for the revised paper. “I believe that the correction and the detailed editor’s note present a robust account of events and that the level of uncertainty and caution now expressed by the authors is appropriate,” she emailed me last week in response to the open letter. “In my view, readers and guideline bodies are now equipped to reach their own conclusions on the quality of the evidence in support of this therapy.”

From my perspective and that of colleagues who signed the open letter, Godlee’s statement represents an abdication of BMJ’s responsibility to safeguard the medical literature from poor quality research. No amount of acknowledged “uncertainty” and “caution” can compensate for this study’s deficiencies. The BMJ imprimatur should serve as something more than an advisory that readers have to “reach their own conclusions” about the integrity of research published in the organization’s journals.

A complicating factor could be that the Lightning Process trial’s lead investigator, Dr. Esther Crawley of Bristol University, has a close relationship with BMJ. Crawley, a pediatrician, has published many papers about this illness in BMJ journals and is an associate editor at BMJ Paediatrics Open. Unfortunately for BMJ, in addition to the mess over the Lightning Process study, Crawley was also directed earlier this year to correct the ethics statements in eleven other published papers — five of them in BMJ journals. (Disclosures from my ongoing Virology Blog investigation triggered this development as well.)

BMJ’s inability to detect problems in these cases, despite supposedly rigorous oversight systems, suggests that its peer-review and editorial processes might need a significant overhaul. Beyond that, the failure to take appropriate steps in the face of disqualifying flaws, like those in the Lightning Process study, suggests that BMJ is prioritizing concerns about reputational damage or other interests over the health and well-being of children, at least in this arena of science.

David Tuller is a senior fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. Members of the ME/CFS patient community generously donated to a UC Berkeley crowdfunding campaign in support of Tuller’s academic position. Esther Crawley, the lead investigator of the Lightning Process study, has made unsupported allegations of libel against Tuller, and UC Berkeley has received complaints about his “actions and behaviour” from Bristol University. UC Berkeley has reviewed and dismissed these complaints.