Reproducibility in Science

Improving transparency could make it easier for scientists to reproduce others' findings. Photo by mrhayata.

Improving transparency could make it easier for scientists to reproduce others’ findings. Photo by mrhayata.

The scientific method relies heavily on the reproducibility of experimental data. In fact, reproducibility implies that using the same reagents and model systems, similar experimental results should be obtained as those reported, even if the laboratories and equipment are different. This aspect of science has become a prominent challenge as the number of papers retracted has been increasing since the first reported retraction in 1977. The inherent problem of irreproducibility is a waste of time and effort of other scientists in following false leads and the resulting controversies, which arise from apparently conflicting results, are due to ambiguity in methodology. Science can advance when results obtained are validated and peers in the field can use those results as building blocks for their future experiments. Lately, too many published results do not hold up to scrutiny and this causes delays in projects based on those results and techniques.

In an effort to quantify this irreproducibility, an experiment was conducted where a (Vasilevsky et al 2013) major cause of the problem could be ascertained. It was hypothesized that some of the inconsistencies stemmed from a lack of “identifiability” of reagents, tools, and model systems used. In this study, methodologies from various articles in a broad range of journals were analyzed and judged on how well the resources used in these studies were reported. Among the resources quantified were model organisms used, cell lines, strains, and reagents (such as antibodies and knockdown reagents such as morpholinos and RNAi oligos). They found that 54% of these resources were not uniquely identifiable, making it difficult for peers to reproduce the exact test conditions. Even though these are details that should be reported in the “Materials and Methods” section of an article they can often be vague, incomplete, and ambiguous. This article posits lack of identifiability as a leading cause of irreproducibility despite the high reporting standards required by journals. Indeed another study performed by the National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3R) reported that only 59% of the 271 articles examined stated the organisms’ specifics such as weight, age, and sex; and 70% lacked details of statistical analyses (Kilkenny et al 2009).

So how do we as scientists not only address this problem but strive to adhere to more stringent regulations in reporting our science and evaluating our peers? This has to be a combined effort from journal editors, reviewers, scientists, and scientific organizations to increase the transparency of the methods used. To this effect, in June 2014, a gathering was held with scientific leaders and various journal administrators concerning principles of conducting and reporting scientific research. A guideline was established promoting journal policies that would support greater clarity in methods. In keeping with these guidelines Nature Publishing Group has established a multi-point reporting checklist, which enumerates the improved reporting standards that should be upheld in science. This checklist includes directions on reporting scientific design, the statistics performed, describing reagents in proper detail, presenting clinical trial data, and the correct deposition of that data, and it also has specifics on gel electrophoresis and blot data presentation. In 2014, ASCB also began a Reproducibility Task Force to examine the issue of reproducibility in life science research, especially in basic research. The Task Force issued a white paper in 2015 expressing 13 recommendations to tighten standards in labs, research institutions, federal agencies, and scientific journals.

Some of the current issues include insufficient data about the nature and size of sample sets and replicates performed. The statistical analysis of the data also implies different conclusions depending on whether replicates are technical or biological and also on the type of statistical test used. Very often the data and graphs will not be representative of the variability in the source data. To address these issues, authors are encouraged to plot the full spread of their data and also provide the source files for data underlying the graphs, especially if the sample size is small. The same can be applied for cytological image based data whose quantification methods must be analyzed closely and stand up to rigorous image verification. The other issue is transparency in reagents and models used. Antibodies and cell lines are often not described in great detail and contribute to irreproducibilty.

The real challenge, however, is in changing the deeply entrenched practices, which engender irreproducibility. This can only be achieved if everyone in the scientific community is aware of the problem and works together to resolve it. The guidelines for promoting rigorous reporting and transparency of data should be adopted by the publishing journals and be included in the author guidelines. In fact recently Nature Cell Biology has pledged to adhere to these strict rules. Also reviewers should be encouraged to scrutinize and request raw data and not accept conclusions from resulting graphs only. Original image files should be required along with gels and blots and made available such that peers can compare their data to the existing raw data preferably in supplemental data files. These reporting checklists will hopefully encourage scientists to report their data responsibly and make it easier to replicate. But first, we all have to recognize that reproducibility is a real problem that needs immediate attention and all our concerted efforts to alleviate it. However it should be borne in mind that some experiments, especially those involving mammalian systems like mice, are extremely expensive and require years to complete; asking for repeats in those cases may not be feasible. Instead we can ask them to be transparent in how they performed the experiments that they did.

About the Author:


Arunika is a post-doctoral researcher in the labs of Drs. Michael Lampson and Ben Black at the University of Pennsylvania. She is working on the mechanism of centromere inheritance and maintenance in the mammalian germline.