Peer review could help smoke out the next Theranos

Over a 25-month period, the startup Theranos plummeted from the heights of biotech, with a valuation of $9 billion and a fawning profile in the New Yorker’s “Annals of Innovation” in December 2014, to thanatos (death) when the company shuttered its final laboratory in January 2017. By early 2018, joint investigations by the Securities and Exchange Commission and the Food and Drug Administration, impending bankruptcy, mass layoffs, and fraud charges against the company’s enigmatic founder, Elizabeth Holmes, finished off the so-called unicorn.

Not long after the sentinel New Yorker profile appeared, one of us (J.P.A.I.) raised serious doubts about Theranos’ operation in a JAMA Viewpoint. One major concern was that the company hadn’t presented a shred of peer-reviewed evidence for its technology. Another issue was the company’s core value proposition: to do more tests on more people, even if inexpensive and “easy” to conduct, thanks to proprietary Theranos technology.

Although this article received pushback from the company, the author insisted that the opacity begotten by “stealth research” was unlikely to facilitate the transparency required to scrutinize such disruptive innovation. Several months later, Wall Street Journal reporter John Carreyrou began methodically illuminating inconvenient truths behind the façade of Holmes’ black turtlenecks and “trade secrets.”

What went wrong with Theranos? In postmortem case studies by business schools, abundant issues with corporate governance, shareholding structure, workplace culture, marketing fraud, and product-centered mirage seemed obvious enough in hindsight. But not before hundreds of thousands of patients had undergone unnecessary needle-sticks and received millions of inaccurate test results.

A ‘case,’ or a trend?

It’s easy to write off Theranos as a uniquely bad apple. But a closer look at other unicorns shows that secrecy is endemic to the rapidly multiplying herd.

That is our conclusion after assessing the participation of health-care unicorns in the peer-reviewed literature. Publishing isn’t the primary mission of startups — it is even sometimes perceived as a deviation from the trajectory of disruptive innovation. Yet when technologies influence real-world health outcomes, peer-reviewed validation is essential for accountability and credibility in the biomedical community.

Today, there are 309 unicorns with an aggregate valuation above $1 trillion. The median venture capital fundraise is $165 million, the highest on record. The dataset in our analysis, published Monday in the European Journal of Clinical Investigation, backdates to November 2017, when there were 18 current and 29 former (leaving the herd by acquisition or public offering) health care unicorns.

We found that most unicorns published very little in the peer-reviewed literature. One-quarter published nothing. Of papers that were published, 1 in 10 were considered high-impact (garnering more than 50 citations by subsequent papers). The majority of current unicorns and 40 percent of exited unicorns had zero high-impact papers. In addition, of eight high-impact current unicorn papers, two were unrelated to the company’s product. Only one of the 34 high-impact papers involved any human subjects, and that was a retrospective observational study.

Given the revolutionary claims made by this cohort of unicorns — ranging from curative immunotherapies and vaccines for cancer to machine-learning enabled reinvention of drug discovery to nanotechnology powered genomic sequencing — their impact in the public literature seems empirically unimpressive, close to nil. Unicorns publishing nothing consequential in the peer-reviewed literature still raised billions of dollars.

Some have argued that scientists who publish prolifically, by serving in leadership roles at unicorns, offer indirect reassurance and credibility via “track records.” To address this claim, we analyzed the composition of executive boards. The majority of unicorns had no such scientific superstars at their helms. And while scientific advisory boards might provide another venue for housing such expertise, information on these boards was rarely publicly available.

Lastly, we noted that matched, lesser-valued competitors (with a mean valuation of $100 million) exhibited an even more marked dearth of publications.

Beyond Theranos: ensuring innovations and preventing illusions

Our analyses showed that the highest-valued health care startups (past and present) contribute minimally to relevant, high-impact published research. Competitors with less funding publish even less. Moreover, company valuations are unrelated to publication records. While unicorns may own many patents, patent appraisal is less rigorous than peer review. Theranos, for example, had more than 100 patents, but these failed to materialize into a real product.

Through independent examination, peer review would help evaluate the relative merits of the proposed disruption. For Theranos, it might have highlighted the core conceptual misunderstanding: Too much unnecessary testing can produce overdiagnosis, false positives, and wasted resources. This alone could have obviated the company’s value proposition.

Allegations of fraud, as were lobbied against Theranos, make headlines. But the common, more insidious problem with health care unicorns is not one-off scandal ripe for cinema. It is the absence of credible scientific evidence. Health care products based solely on data generated internally may be problematic and untrustworthy.

Confidential data sharing cannot replace open scrutiny by the scientific community. Startups are key purveyors of innovation: Holding them to a minimum standard of evaluation is essential. Peer review, while imperfect, is indispensable for validating novel technologies in biomedicine. As Aaron Carroll wrote in the New York Times, peer review is “the worst method to judge research, except for all the others.”

In contrast to other sectors, where the virtues of products readily emerge or fail in the marketplace, it takes deliberate, objective study to codify health outcomes. In our article, we suggest a variety of low-resource strategies that unicorns can leverage to fulfill expectations of transparency via publication. These include validation of products without divulgence of technical details; contribution of technologies to the efforts of external research groups; or collaboration with academic institutions.

Startups, of course, worry about being scooped. Yet patents protect commercial rights, allowing startups to participate in peer review without fear of idea theft. There is no need for numerous papers. Discrete pivotal, high-impact articles would suffice to document revolutionary technologies.

From the policy perspective, our results call for caution adopting health care products from entrepreneurial companies that provide little objective documentation. Investors should reassess their participation in health care ventures that are thin in published scientific evidence.

To the extent that health care startups are responsible for pushing the frontiers of medicine, patients’ lives hang in the balance. Hope and hype cannot supplant data.

Eli Cahan is a medical student at the New York University School of Medicine between third and fourth year beginning a master’s in health policy at Stanford University in September. Ioana A. Cristea is an associate professor at Babes-Bolyai University in Romania, a research affiliate at the Meta-Research Innovation Center at Stanford (METRICS). John P.A. Ioannidis is the chair in disease prevention, professor of medicine, health research and policy, biomedical data science, and statistics at Stanford University, and co-director of METRICS.