Over a 25-month period, the startup Theranos plummeted from the heights of biotech, with a valuation of $9 billion and a fawning profile in the New Yorker’s “Annals of Innovation” in December 2014, to thanatos (death) when the company shuttered its final laboratory in January 2017. By early 2018, joint investigations by the Securities and Exchange Commission and the Food and Drug Administration, impending bankruptcy, mass layoffs, and fraud charges against the company’s enigmatic founder, Elizabeth Holmes, finished off the so-called unicorn.
Not long after the sentinel New Yorker profile appeared, one of us (J.P.A.I.) raised serious doubts about Theranos’ operation in a JAMA Viewpoint. One major concern was that the company hadn’t presented a shred of peer-reviewed evidence for its technology. Another issue was the company’s core value proposition: to do more tests on more people, even if inexpensive and “easy” to conduct, thanks to proprietary Theranos technology.
Although this article received pushback from the company, the author insisted that the opacity begotten by “stealth research” was unlikely to facilitate the transparency required to scrutinize such disruptive innovation. Several months later, Wall Street Journal reporter John Carreyrou began methodically illuminating inconvenient truths behind the façade of Holmes’ black turtlenecks and “trade secrets.”
What went wrong with Theranos? In postmortem case studies by business schools, abundant issues with corporate governance, shareholding structure, workplace culture, marketing fraud, and product-centered mirage seemed obvious enough in hindsight. But not before hundreds of thousands of patients had undergone unnecessary needle-sticks and received millions of inaccurate test results.
A ‘case,’ or a trend?
It’s easy to write off Theranos as a uniquely bad apple. But a closer look at other unicorns shows that secrecy is endemic to the rapidly multiplying herd.
That is our conclusion after assessing the participation of health-care unicorns in the peer-reviewed literature. Publishing isn’t the primary mission of startups — it is even sometimes perceived as a deviation from the trajectory of disruptive innovation. Yet when technologies influence real-world health outcomes, peer-reviewed validation is essential for accountability and credibility in the biomedical community.
Today, there are 309 unicorns with an aggregate valuation above $1 trillion. The median venture capital fundraise is $165 million, the highest on record. The dataset in our analysis, published Monday in the European Journal of Clinical Investigation, backdates to November 2017, when there were 18 current and 29 former (leaving the herd by acquisition or public offering) health care unicorns.
We found that most unicorns published very little in the peer-reviewed literature. One-quarter published nothing. Of papers that were published, 1 in 10 were considered high-impact (garnering more than 50 citations by subsequent papers). The majority of current unicorns and 40 percent of exited unicorns had zero high-impact papers. In addition, of eight high-impact current unicorn papers, two were unrelated to the company’s product. Only one of the 34 high-impact papers involved any human subjects, and that was a retrospective observational study.
Given the revolutionary claims made by this cohort of unicorns — ranging from curative immunotherapies and vaccines for cancer to machine-learning enabled reinvention of drug discovery to nanotechnology powered genomic sequencing — their impact in the public literature seems empirically unimpressive, close to nil. Unicorns publishing nothing consequential in the peer-reviewed literature still raised billions of dollars.
Some have argued that scientists who publish prolifically, by serving in leadership roles at unicorns, offer indirect reassurance and credibility via “track records.” To address this claim, we analyzed the composition of executive boards. The majority of unicorns had no such scientific superstars at their helms. And while scientific advisory boards might provide another venue for housing such expertise, information on these boards was rarely publicly available.
Lastly, we noted that matched, lesser-valued competitors (with a mean valuation of $100 million) exhibited an even more marked dearth of publications.
Beyond Theranos: ensuring innovations and preventing illusions
Our analyses showed that the highest-valued health care startups (past and present) contribute minimally to relevant, high-impact published research. Competitors with less funding publish even less. Moreover, company valuations are unrelated to publication records. While unicorns may own many patents, patent appraisal is less rigorous than peer review. Theranos, for example, had more than 100 patents, but these failed to materialize into a real product.
Through independent examination, peer review would help evaluate the relative merits of the proposed disruption. For Theranos, it might have highlighted the core conceptual misunderstanding: Too much unnecessary testing can produce overdiagnosis, false positives, and wasted resources. This alone could have obviated the company’s value proposition.
Allegations of fraud, as were lobbied against Theranos, make headlines. But the common, more insidious problem with health care unicorns is not one-off scandal ripe for cinema. It is the absence of credible scientific evidence. Health care products based solely on data generated internally may be problematic and untrustworthy.
Confidential data sharing cannot replace open scrutiny by the scientific community. Startups are key purveyors of innovation: Holding them to a minimum standard of evaluation is essential. Peer review, while imperfect, is indispensable for validating novel technologies in biomedicine. As Aaron Carroll wrote in the New York Times, peer review is “the worst method to judge research, except for all the others.”
In contrast to other sectors, where the virtues of products readily emerge or fail in the marketplace, it takes deliberate, objective study to codify health outcomes. In our article, we suggest a variety of low-resource strategies that unicorns can leverage to fulfill expectations of transparency via publication. These include validation of products without divulgence of technical details; contribution of technologies to the efforts of external research groups; or collaboration with academic institutions.
Startups, of course, worry about being scooped. Yet patents protect commercial rights, allowing startups to participate in peer review without fear of idea theft. There is no need for numerous papers. Discrete pivotal, high-impact articles would suffice to document revolutionary technologies.
From the policy perspective, our results call for caution adopting health care products from entrepreneurial companies that provide little objective documentation. Investors should reassess their participation in health care ventures that are thin in published scientific evidence.
To the extent that health care startups are responsible for pushing the frontiers of medicine, patients’ lives hang in the balance. Hope and hype cannot supplant data.
Eli Cahan is a medical student at the New York University School of Medicine between third and fourth year beginning a master’s in health policy at Stanford University in September. Ioana A. Cristea is an associate professor at Babes-Bolyai University in Romania, a research affiliate at the Meta-Research Innovation Center at Stanford (METRICS). John P.A. Ioannidis is the chair in disease prevention, professor of medicine, health research and policy, biomedical data science, and statistics at Stanford University, and co-director of METRICS.
There is little difference between the content marketing we see every day on sites like this, or our mass media news. There are plenty of places to get unscientific hype published for a price. the general publis has no accsess to the peer review process, and for a price anyone can get credible appearing peer review. There are plenty of companies acting just as agregios as Theranos, they are just more aware of the wording they use to manipulate investors or the public. After all no peer review goes into any of the science writing that is passed off in mass media, in order to generate buzz. We wont see the FTC cracking down on misiinformaion in content marketing or the business pages, where most of these cons are publicized.
But see this article, peer review is a game
Of potential verification to you point(s), also consider the online lecture, from her published book THE TRUTH ABOUT THE DRUG COMPANIES by Dr. Marcia Angell, MD, Harvard Medical School and retired Editor and Chief of The New England Journal Of Medicine.
Hoaxes and fakes and the Death of Truth are the monsters of our time.
Has truth died?? Are we en route to more Totalitarian regimes??
Our electric informers are a hazard
” A Unicorn Is A Mystical Beast” James Thurber
There are two additional factors working to support biopharma unicorns that will also need to be dealt with:
1) Fear of Missing Out – Companies will invest in companies like this because they are afraid of missing out on the next great thing, and they don’t want to be scooped by their competitors. There were two large corporations (Safeway and Walgreens) in the case of Theranos that demonstrated a shocking lack of due diligence in looking at what they were buying. They should be punished by their shareholders for this behavior.
2) Greater fool theory. As long as people can make money propping up and selling these dogs, they won’t be deterred by a lack of peer review of the science. Peter Thiel’s Stemcentryx and it’s Rova-T (a huge implosion for Abbvie) comes to mind when I think of this category.
Some people just don’t want to let facts and data get in the way of a good story where there is serious money to be made.
Very true, and to the point…see the same cyclical hype going on even in established public securities, to move puts, or call, volumes, or riduculouse P/W raruos, with no connection to basic facts. Elon Musk’s quip, on the SEC, being the ‘Shortsellers Enrichment Cabal’ is not far off the mark.
Typos, thanks to spell correct, I meant ‘ridiculous P/E ratios’
All one needed to know about Theranos was already there; people just wanted to be fooled by a great backstory. I first heard of Theranos around 2013. I looked for any form of validated data and there wasn’t any. To be honest I thought that it would be forthcoming but the story had a different ending. The answer is in your story — one can invest in data-free unicorns but take the blame if the company is a bust. No investor in Theranos has anyone to blame other than themselves.
Caveat Emptor (Let The Buyer Beware) does indeed partially apply, however, Theranos was only one of many VC fraudams and scams about the sane time, like Zenefits, Hampton Creek, and Soylent Bars, curiously all from the same Ventura Capital backers as Theranos….
Bernie Madoff got caught but these VC’s are equally fraudsters and crooks.
If she hadn’t been a cutesy blonde with a Stanford background who dressed like Steve Jobs she’d have never got a dime
Follow the money, the venture capitalists are the guilty parties, with their corrupt term sheets and pump and dunp insider trading fraud models.
Peer review publications are pointless. And indeed, often used to validate scams.
Investors should simply do their homework, and possibly follow advice from qualified consultants. That’s assuming there are actually such professionals, as often that doesn’t seem to be the case. Surely, there are plenty of consultants pretending to know what they are doing, out there.
As a skilled, knowledgeable scientist, with the hobby to dig behind the claims of flashy startups, I can safely say that the sector is riddle with questionable claims, to downright biomedical/scientific scams.
Anyway, if sound science would be the only metrics to be taken in to account, 90+% of startups wouldn’t make the cut. But, what investors really care to know is: can they make money? Regardless, any therapeutic or scientific validity.
Amarin, Sage Ther., Aimmune Ther. are just a few examples of that
Comments are closed.