A core concept in scientific research is that empirical results must be replicable. This concept dates back to the birth of the experimental method itself. The Accademia del Cimento (Academy of Experiment) was founded in Florence in 1657 by Galileo’s students and it published the first manual of scientific experimentation, a guide for data collection and methodological standardization. The society’s motto was provando e riprovando (trying and trying again), emphasizing the importance of replication of scientific experiments.
Fast forward to the present day where scientific discovery proceeds at an impressive pace and yet we find that in many instances research findings cannot be replicated.1,2 The causes for the lack of replication have been examined, revealing a complex scenario with multiple determinants3 ranging from sheer sloppiness (which is inexcusable) to the almost Twitter-length restrictions imposed on the materials and methods sections of many glamorous journals. Other culprits implicated include selection bias in publishing only positive results and the hypercompetitive quest for scientific discoveries that forces scientists toward sensationalism in presenting their results. It is important to note here that I am not talking about fraud. That is a wholly different issue.
So let me also state that there is no shortcut to rigor in science and to accurate communication of its results. It is imperative that we have in place all the checkpoints that ensure rigorous and reliable published literature. After all, this is the purpose of the lengthy and thorny process of scholarly publishing.
The topic of irreproducibility is receiving a lot of attention in both the specialized and the lay press. Some of the contributions can be very sensational and receive a lot of attention, but frankly offer little of practical relevance or are simply wrong. Others may be less flashy, but can contribute to solving this serious problem. Let me start with an egregious example in the first bucket and then move on to what I find to be of help.
Leaving a Wrong Impression
A paper by Freedman et al.4 published recently in PLoS Biology could give the wrong impression to those who have not followed this problem at a granular level. The authors attempt to estimate the economic implications of the lack of reproducibility in the life sciences. The paper concludes that data irreproducibility has a huge economic cost, $28 billion, or over half of all money spent on preclinical research every year in the United States. While the paper does not say that the money is “wasted,” I believe that many, including some in Congress, will come away from this analysis with that word in mind. Nothing could be further from the truth than the notion that $28 billion is wasted every year on useless preclinical research.
First of all, by reading the supplementary material, one finds out that the paper lumps together four very different categories of irreproducibility (in the areas of study design, biological reagents, protocols, and reporting) and then estimates that ~50% of the published papers fall into one of these categories. The authors then look at the total amount of spending in research and development and decide that ~50% of that is associated with irreproducibility.
According to this analysis, the fact that one investigator needs to change the pH from 7.2 to 7.5 to grow a finicky cell line would mean that the entire cost for this research is “irreproducible” and thus a waste of money. No, sorry. This does not make any sense and it does not pass any reasonable test of validity.
Let me offer an example. Say we download a new version of some software, a beta version perhaps. A few months later, we have to download an updated version of the same software, because users found bugs. This often happens with iPhone software, for example.
“[I]f the new iPhone software needs tweaking, is it legitimate to conclude that the iPhone is an economic and technological failure?”So by the same logic, if the new iPhone software needs tweaking, is it legitimate to conclude that the iPhone is an economic and technological failure? Modification and refinement are integral to innovation, and especially so in the life sciences where we are constantly working at the frontier between what is known and what is still unknown.
Do not take my argument as an apology for bad science. I am all for rigor and for straightening out this very important problem. However, this sensational and broad-brush analysis of a complex issue is not helpful. It is not a constructive contribution toward finding solutions to the problem.
Increasing Rigor in Science
ASCB has taken a different approach to the problem of irreproducibility. It recently released a white paper titled “How can scientists enhance rigor in conducting basic research and reporting research results?” (www.ascb.org/reproducability/).
“[T]he problem of irreproducibility is real in basic science and in cell biology as well as in preclinical research.”This paper is the work of a committee chaired by Mark Winey from the University of Colorado, Boulder, which tackled the issue for over a year, conducted a survey of the ASCB membership, and worked with the ASCB Council to put forward some key recommendations to increase rigor in science in general and in cell biology in particular.
First of all, the qualitative survey of the ASCB membership confirms that the problem of irreproducibility is real in basic science and in cell biology as well as in preclinical research. Second, the paper offers a multi-tier definition of reproducibility, which is an essential step because of the very different determinants that confound the problem. Teasing apart the various causes of irreproducibility is essential to help us think of solutions to the problem and to avoid sensational sweeping statements or policies that could inflict more damage than they provide solutions.
And third, the paper offers some important recommendations for improving the situation and solving problems. I encourage you to read the paper, since it would be too long to discuss every recommendation here. However, suffice it to say that ASCB identifies some key areas for action:
- Emphasize the importance of training and mentorship, both generally and in statistics, to enhance rigor in science
- Further disseminate the San Francisco Declaration on Research Assessment (DORA)5 to limit the pressure on authors to sensationalize research results so they can publish in high–impact factor journals
- Encourage publishing practices such as reviewer checklists, citations of open source repositories, and sharing of primary data, protocols, and materials
- Work with other groups and promote community venues to develop and define appropriate standards that improve scientific outcome without imposing excessive burden on scientists
Another constructive proposal emerged recently. A group of scientists published an article in Science proposing author guidelines to promote transparency, openness, and reproducibility.6 Although papers and initiatives of this kind definitely fall into the less-glamorous category, they certainly offer the most effective course to take to solve this complex problem. For this reason, ASCB has signed on to the Transparency and Openness Promotion (TOP) guidelines to inspire its scientists and its scholarly journals to think about how they can help solve the problem.
Science ultimately is a self-correcting process, often curvy and not as fast as we would like it to be.
“As science grows more sophisticated, getting reproducibility right is a critical challenge.”But just as in creating software, there are costs involved in upgrading scientific knowledge, including the publication of bug-ridden, imperfect, barely working, and not fully understood results that nonetheless point researchers toward valuable discoveries. As science grows more sophisticated, getting reproducibility right is a critical challenge. ASCB’s job, just like that of the Accademia del Cimento in the 17th century, is to put forward ideas to the field and, as a publisher of important journals, to ensure that scientists remain authoritative in their scholarly communications. We must facilitate the scientific process and its self-correction as effectively as possible.
1Begley CG, Ellis LM (2012). Raise standards for preclinical cancer research. Nature 483, 531–533.
2Prinz F et al. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10, 712.
3Collins FS, Tabak LA (2014). Policy: NIH plans to enhance reproducibility. Nature 505, 612–613.
4Freedman LP et al. (2015). The economics of reproducibility in preclinical research. PLoS Biol 13, e1002165.
6Nosek BA et al. (2015). Scientific standards. Promoting an open research culture. Science 348, 1422–1425.