Giant pandas. Self-fertilizing plants. People.
Lots of species have been proposed as “evolutionary dead ends” (though whether they actually are is another question). And now added to those ranks might be science itself.
A new paper has argued that science appears to be in something of an evolutionary cul-de-sac, mired in poor methodology and misguided objectives that have changed only for the worse over the past 60 years. Indeed, science is doing pretty much what it has evolved to do — produce a steady stream of plausible and attention-grabbing but frequently false findings.
At least, that’s what a pair of researchers at the University of California, Davis, said in their new paper, titled “The Natural Selection of Bad Science.” The authors, Paul Smaldino and Richard McElreath, argued that “some of the most powerful incentives in contemporary science actively encourage, reward, and propagate poor research methods and abuse of statistical procedures.”
Smaldino and McElreath, who posted their article on the preprint server arXiv, meaning it has not yet been through peer review, noted that the behavioral sciences in particular fail to produce statistically rigorous studies. The term of art here is “power” — an indication of the ability of an analysis to detect the outcomes it purports to find. And, according to Smaldino and McElreath, “despite over 50 years of reviews of low statistical power and its consequences, there has been no detectable increase” in the average power of studies.
Why the lack of improvement? The UC Davis researchers, like many others, believe that science is simply behaving the way it has evolved to behave: as a publishing factory. “An incentive structure that rewards publication quantity will, in the absence of countervailing forces, select for methods that produce the greatest number of publishable results. This in turn will lead to the natural selection of poor methods and increasingly high false discovery rates.” Sticking with the Darwinian metaphor, Smaldino and McElreath said that successful labs churn out “progeny” — grad students, junior faculty, future lab heads — that mimic their approach.
And the evolutionary metaphor is telling in another way. Evolutionary theory talks about the presence of “cheaters” in any population, but Smaldino and McElreath pointed out that the influence of such cheaters is even more insidious than one would think. For example, cheaters may invent ways to game the system, but others are quickly forced to mimic their methods, lest they be left behind.
A sad fact of science publishing is that some cheaters will always be able to produce bogus papers — at least for a while. Fake data can be tough to spot, particularly if they’re plausible enough at first glance.
What’s needed, then, is a way to improve the “selective” forces at work on the field of science. “All of the incentive structures — the framework of science — really do nudge us toward this kind of research and this kind of reporting, and the less-than-ideal reports of science in the public sphere,” said Tim Caulfield, of the Health Law Institute at the University of Alberta, Canada. “Whether the actions are intentional is irrelevant.”
Caulfield, author of “Is Gwyneth Paltrow Wrong About Everything?”, sees what he calls “the invisible hand of hype” — everything from the journals that promote sexy research over more stolid, but necessary, replication studies to the universities that churn out overwrought press releases — pushing science, and consumers of science news, toward the morass.
And lately, he has noticed that some scientists are objecting to critiques, fearing that what they see as science-bashing could further damage the institution. They’re wrong. “It’s not science-bashing,” Caulfield said. “It’s wrapping our arms around science and saying we need to protect it.”
Still, statistician Andrew Gelman, of Columbia University, said it may not be too late for that next generation of researchers to clean up their act. The new paper, wrote Gelman, could help some scientists “consider the possibility that they’re stuck in a dead end.” There’s only one way out: Evolve!
