Contribute Try STAT+ Today

Few journals have been more admirable than The BMJ (formerly the British Medical Journal) and some of its sister publications under the BMJ brand in highlighting issues of direct significance to health care consumers. So it is baffling — and troubling — when BMJ editors fail to take appropriate action to address unacceptable lapses in high-profile research they have published.

For years, the reading list for my journalism class on public health and medicine at the University of California, Berkeley, included groundbreaking articles in The BMJ on “disease-mongering” — how pharmaceutical companies have manipulated and misrepresented research data to expand existing diagnostic categories and create new ones. I have also appreciated BMJ’s campaign for open access to trial data and its forays into investigative journalism.

Much of this hard-hitting approach can be attributed to Dr. Fiona Godlee, the Cambridge University-educated physician who has led the organization since 2005. Godlee is both editorial director of BMJ, which publishes dozens of titles, and editor-in-chief of The BMJ, one of the world’s leading medical journals. A 2016 profile of Godlee in STAT called her a “crusading editor” who “aims to shake things up in science.” The BMJ, she told STAT, is “a campaigning journal.”

advertisement

That’s why I am disappointed at how BMJ and Godlee have handled a seriously problematic paper in a field I know well — behavioral and psychological interventions for the illness (or cluster of illnesses) often called chronic fatigue syndrome (CFS) but also known as myalgic encephalomyelitis (ME), CFS/ME, and ME/CFS, among other names.

ME/CFS, as U.S. government agencies now refer to it, is a disabling disease marked by immunological, neurological, and other physiological dysfunctions. There are no currently identified diagnostic tests or approved drug treatments for it. The symptoms include a prolonged relapse following minimal amounts of exertion, cognitive impairments, and sleep disorders, among others.

advertisement

For many years, the standard of care mainly involved psychotherapy and a graded exercise program. These treatments were based largely on the widely dismissed theory that patients’ ongoing symptoms were caused by deconditioning, a belief promulgated for decades by a group of U.K. psychiatrists and other health professionals. In the face of serious scientific questions about the research behind this rehabilitative approach, the Centers for Disease Control and Prevention dropped its recommendations for these treatments in 2017.

As a journalist and public health academic at Berkeley, I have written extensively about this issue, largely for a science site called Virology Blog, but also for the New York Times, Undark, and STAT, among others. My investigation has documented substantive methodological and ethical problems with a number of behavioral and psychological studies, some of them published in BMJ journals, suggesting that BMJ, like much of the British academic and medical establishment, might have a blind spot in this domain.

A striking example is a study of a commercial three-day self-help program called the Lightning Process as a treatment for pediatric CFS/ME (as the investigators called the illness). The Lightning Process is a goulash of neuro-linguistic programming, life coaching, osteopathy, positive affirmations, and other elements. It aims to train people to improve their health by altering their thought processes and refusing to allow their symptoms to control them.

Its founder, osteopath Phil Parker, previously taught aspiring healers how to diagnose people’s ailments through auras, tarot cards, energy fields, and the spinning of pendulums, as well as how to communicate with spiritual guides. Lightning Process practitioners have claimed to successfully treat people with multiple sclerosis, eating disorders, and other illnesses. U.K. advertising authorities have cited practitioners for making unsubstantiated medical claims.

The Lightning Process trial was not blinded and relied on self-reported outcomes — a design with enormous potential for bias, since participants’ knowledge of what treatment they received can influence their subjective responses to questions about their health. Nevertheless, based on their findings, the investigators declared that the Lightning Process, in combination with medical care, was an “effective” treatment for pediatric CFS/ME. Archives of Disease in Childhood, a well-regarded BMJ journal, published the results in September 2017.

Beyond questions about the Lightning Process as a treatment modality, the study itself violated core principles of scientific inquiry. The investigators recruited more than half of the participants before trial registration, swapped primary and secondary outcomes after gathering data from the early recruits, and then failed to disclose these critical details in the paper.

I noted these irregularities in a blog post in December 2017, three months after the study was published.

Recruiting participants before the trial was officially registered should have doomed the prospects for publication from the start. In 2005, the International Committee of Medical Journal Editors adopted a policy of rejecting trials in which any participants were recruited before formal registration in a recognized database.

Prospective trial registration requires investigators to provide in advance full details about study design: number of participants, the interventions, primary and secondary outcomes, and other details. Editors of the major journals agreed to this policy as a way of preventing selective outcome reporting — the tendency to extract and highlight the best results rather than sticking to a predesignated plan.

BMJ has been a leading voice for the effort to tighten prospective registration requirements. At a hearing of the U.K. Parliament in 2013, Godlee touted the publisher’s zero-tolerance policy. “As far as we are aware, in the last two years we have not published any trial that has not been prospectively registered,” she declared. In 2015, an editorial in The BMJ called the prospective registration policy “the single most valuable tool we have to ensure unbiased reporting of research studies.”

The Lightning Process study demonstrates what can happen when that “valuable tool” is abandoned. In the trial, the initial primary outcome was school attendance at six months; self-reported physical function was designated a secondary outcome. But after gathering data from children recruited before the trial was registered, the investigators made self-reported physical function the primary outcome and demoted school attendance to a secondary outcome. That bolstered their published findings because they were able to report positive results for self-reported physical function, their revised primary outcome. In contrast, school attendance at six months showed no improvement — a finding that was downplayed and essentially ignored. In other words, the mid-trial switch made it easier for the investigators to assert that the Lightning Process was effective.

By falsely presenting the study as a prospective trial with pre-designated assessment methods, the investigators misled the public, medical practitioners, and policymakers about their results. News accounts amplified this distorted message.

In January 2018, Archives of Disease in Childhood agreed to look into the matter. After a year and a half, the journal finally reissued the paper in July 2019 with an extensive correction along with a reiteration of the original results — that the Lightning Process is “effective.” An accompanying editor’s note sought to justify republishing the study with the same findings rather than retracting it.

The journal’s justifications for this decision cannot withstand scrutiny.

First, according to the editor’s note, “BMJ policy requires prospective registration of randomised trials but we do not consider a failure to enforce that policy grounds for retraction.” The note did not explain the reasoning behind this perplexing conclusion, which plainly conflicts with BMJ’s categorical declarations about the importance of prospective registration.

The editor’s note also tried to massage away the problem of the mid-trial outcome-swapping. The investigators had attributed the change to preferences expressed by early trial participants. According to the editor’s note, the journal sought “assurance from the authors that the change in primary outcome was not influenced by (positive) findings” during the first recruitment phase.

Asking those who might have been influenced by bias if they were, in fact, influenced by bias seems to willfully ignore the central point. Designing studies to minimize bias is important precisely because humans are unreliable assessors of whether our desires and beliefs have affected our decision-making. Such an “assurance” of lack of bias should not be taken at face value, especially when a study’s design itself promotes bias and the actions in question clearly improve a study’s reported results.

Neither the correction nor the editor’s note provided an adequate explanation for why the investigators did not share these salient facts in the published paper. Perhaps the journal decided that question was better left untouched. After all, the U.K.’s Medical Research Council, a major funding agency, defines research misconduct to include “misrepresentation of data, for example suppression of relevant findings and/or data, or knowingly, recklessly or by gross negligence, presenting a flawed interpretation of data.”

The Lightning Process investigators presented a “flawed interpretation of data” by not revealing what they actually did. Whether or not this “suppression” was done “knowingly,” it was certainly done “recklessly” and/or by “gross negligence.” Even if opinions differ on whether this case constitutes actual research misconduct, it is hard to understand how a team of experienced investigators could have omitted such relevant information and offered such an inaccurate account for publication.

Under the circumstances, continuing to grant the Lightning Process study the BMJ seal of approval can only be described as a form of “gross negligence” on the editorial side. In response to the correction, I appealed to Godlee in an open letter of concern in August 2019, signed by 55 scientists, academics, and other experts from Harvard, Columbia, Stanford, University College London, UC Berkeley, Queen Mary University of London, and elsewhere.

The letter called Archives of Disease in Childhood’s decision “scientifically and ethically indefensible” and “potentially harmful” to children, noting that the journal “has now re-published positive primary results generated by outcome-swapping in a study that did not meet a strict publication requirement [i.e. prospective registration] specifically intended to prevent outcome-swapping and other kinds of selective outcome reporting.” With no response from Godlee, I sent the letter again in November, this time endorsed by even more experts and over 60 patient and advocacy organizations from around the world.

Some of the signers also emailed Godlee directly to express dismay. “As you are aware, many other scientists of repute have signed on to the original letter to you,” wrote Vincent Racaniello, a professor of microbiology at Columbia and the host of Virology Blog. “I am sure we could have hundreds more. How many do you need before you are convinced that the work should be retracted? How much longer do you wish to risk harm to young patients?”

Dr. Ronald Tompkins, a professor of surgery at Harvard Medical School, informed Godlee that he was “appalled” that the study remains in the literature. And Brian Hughes, a psychology professor at the National University of Ireland, Galway, wrote Godlee that the journal’s apparent rewarding of questionable conduct “will strongly encourage other researchers to play fast and loose with pre-registration, in the knowledge that their findings can still be published in a BMJ journal.”

Despite such vigorous appeals, Godlee has now reaffirmed BMJ’s support for the revised paper. “I believe that the correction and the detailed editor’s note present a robust account of events and that the level of uncertainty and caution now expressed by the authors is appropriate,” she emailed me last week in response to the open letter. “In my view, readers and guideline bodies are now equipped to reach their own conclusions on the quality of the evidence in support of this therapy.”

From my perspective and that of colleagues who signed the open letter, Godlee’s statement represents an abdication of BMJ’s responsibility to safeguard the medical literature from poor quality research. No amount of acknowledged “uncertainty” and “caution” can compensate for this study’s deficiencies. The BMJ imprimatur should serve as something more than an advisory that readers have to “reach their own conclusions” about the integrity of research published in the organization’s journals.

A complicating factor could be that the Lightning Process trial’s lead investigator, Dr. Esther Crawley of Bristol University, has a close relationship with BMJ. Crawley, a pediatrician, has published many papers about this illness in BMJ journals and is an associate editor at BMJ Paediatrics Open. Unfortunately for BMJ, in addition to the mess over the Lightning Process study, Crawley was also directed earlier this year to correct the ethics statements in eleven other published papers — five of them in BMJ journals. (Disclosures from my ongoing Virology Blog investigation triggered this development as well.)

BMJ’s inability to detect problems in these cases, despite supposedly rigorous oversight systems, suggests that its peer-review and editorial processes might need a significant overhaul. Beyond that, the failure to take appropriate steps in the face of disqualifying flaws, like those in the Lightning Process study, suggests that BMJ is prioritizing concerns about reputational damage or other interests over the health and well-being of children, at least in this arena of science.

David Tuller is a senior fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. Members of the ME/CFS patient community generously donated to a UC Berkeley crowdfunding campaign in support of Tuller’s academic position. Esther Crawley, the lead investigator of the Lightning Process study, has made unsupported allegations of libel against Tuller, and UC Berkeley has received complaints about his “actions and behaviour” from Bristol University. UC Berkeley has reviewed and dismissed these complaints.

  • The Lightening Process does more harm than anything, and using it in pediatric medicine is child abuse. The BMJ really needs to rethink its stance on this, loudly so the people in the back can hear.

  • Thanks, David for this. As an ME/CFS patient it is rather terrifying to learn that BMJ is allowing publication of flawed research which GP’s might use as a guide for patient treatment.

  • Godlee & the BMJ appear to be stuck in the Middle Ages. The Psychiatric Lobby overtly espouses ‘non-material’ causes of illness, very much like mediaeval religion & faith cures – which made a lot of money for the Church at that time.
    Gradually even pure ‘mental’ diseases, like Depression, are gradually being accepted as having a biomedical cause (gut flora, air pollution). But the fans of ‘programming’ as a cause still hold out with ME & allied illnesses, despite the reams of evidence.
    Not content with mediaeval medicine, the BMJ under Godlee also appear to be in favour of scientific cheating to try and prove their mediaeval theories.

  • My grandson is a sufferer of ME/CFS and (diagnosed fibromyalgia). The hospital clinic to which he was referred in Brisbane touted and followed the results of this trial (Lightening Process). Despite high quality results from Griffith University in the same state (Gold Coast) on the immunological parameters of this illness. The effects of such misrepresentation has effects far and wide from the original source. In particular the BMJ journals are read by many general practitioners who would never read a journal such as Frontiers in Immunology.

  • This really makes me wonder if someone is getting paid off. To continue to publish the false information is disgusting and a disgrace. Why is the BMJ putting its reputation in the blender?

  • It is human to make errors. On important matters we put safeguards in place to minimise this likelihood. When safeguards fail we need to act swiftly, determine why and remedy.
    Why would you not do this ? Especially when it’s involving children and health, in addition to compromising the validity/value of research (your business stream).

  • It is extremely disturbing that abject dereliction of scientific process, norms and accountability is acceptable to any BMJ journal, let alone its facilitation. From both medical and legal standpoints, one would not expect that from supposedly ‘lesser’ journals.

    BMJ journals, including the Lancet, have made themselves curiously instrumental in creating ridicule of the medical profession in scientific circles on the subject of this illness. It has almost become a rule that progress in scientific research into ME/CFS a lack of involvement by medics. “Achieving” this retrograde step has required determination.

    The damage goes beyond immediate harm for sufferers and unwitting medical frontline ignorance. It undermines the very role of the scientific method in determining medical practice and norms. This is a reversal of previous progress and discipline that is measurable in centuries – an indictment with grave implications that every field of medicine will be grateful to see rectified.

  • Excellent article. Extremely important. Journals don’t seem to understand the potential harm. My ME was made much worse by exceeding my energy limitations (which would happen to anyone doing LP) I went from working full time to housebound and again from housebound to bedbound.

  • Thankyou for publishing Tullers article on this. It’s deeply concerning that a questionable intervention was trialled on children, in addition to the questionable issues with the whole ethics process of the study.

  • The ethical lapses in this trial are particularly upsetting because they involve an especially vulnerable patient population: ill children.

    It is widely known that the cardinal symptom of ME/CFS is post-exertional malaise or the worsening of symptoms following exertion. This paper does not mention or discuss that symptom or if it even existed in any of the children. Why? I can think of 3 possibilities: 1) If they documented that exertion made the children worse and LP encourages them to push themselves, they would knowingly be putting the patients into harm’s way. 2) By ignoring PEM as required criteria for the study, they are able to recruit children who don’t have ME/CFS, but instead have some type of behavior which may be influenced by LP. 3) They just didn’t know any better.

    Unfortunately, the lack of research funding for this disease has left a vacuum which is too often filled by unproven pseudo-science touting “cost-effective” fixes for a condition we have yet to fully understand. Desperate patients such as those with ME/CFS are frequently targeted into these get-well-quick schemes. By continuing to allow publication of these ethically-challenged studies that use weak-evidence to support their claims, the BMJ is setting a precedence in which more similar studies and practices thrive.

Comments are closed.

Your daily dose of news in health and medicine

Privacy Policy