The challenges of reproducible research

Last Updated on March 11th, 2019 by Sharon Johnatty

The issue of irreproducible research has been of concern to scientists for decades, but recently there has been considerable focus on this problem by leading journals in various commentaries and editorials, and calls for recognition and response to this problem by the scientific community.

Pressure to publish has dominated academic research for over half a century, and policies that inadvertently reward quantity at the expense of quality, and the rising focus on citation counts and other metrics that are increasingly used as a proxy for impact, have contributed to concerns that we are drowning in false or exaggerated claims. Science publishing has mushroomed into a global industry with new players aiming for market share in a climate that values publication numbers as a marker of success and a currency for academic advancement.

The exponential growth in numbers of publications and higher numbers of citations of articles, regardless of the quality has led to the belief that we are in a highly productive era. A recent report of the most influential biomedical researchers identified more than 15 million authors of more than 25 million scientific papers published in between 1996–2011. Analysis of publication patterns of over 40,000 researchers who published two or more papers in the first fifteen years or their ‘early-career’ phase, found that research productivity had not increased for most disciplines, taking into account co-authorship.

It is not an easy decision to know when to publish, and as mentioned in our recent blog, applying the ‘Goldilocks’ rule and good scientific citizenship could save a career. There should be no issue with publishing preliminary hypothesis-generating work that is accessible to the research community. The problem is when these findings are publicized to the lay community as having clinical value, particularly for diseases with heavy burdens in poorer populations that raise false hope. Disseminating findings that prove to be meaningless and is reversed over time could have a ‘crying wolf’ effect and undermine public trust in science. 

“More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” (Nature 2016)

It has become increasingly evident that much of the published literature has findings that cannot be reproduced. Reproducibility separates the anecdotal from the real results that should be able to withstand the test of time and replication by the same or other researchers. Nature recently conducted a survey of 1576 researchers and found that more than 50% of researchers could not reproduce their own experiments, and more than 70% could not reproduce other researchers’ experiments. In this survey, more than 60% of respondents cited pressures to publish and selective reporting to be the main contributors to irreproducibility; more than 50% cited low statistical power and fewer still cited variability in reagents and techniques that were difficult to replicate. An overarching problem was also lack of sufficient time to plan and execute protocols, and senior lab members with limited time to train and mentor junior researchers.  Given the importance of mentoring in research, junior researchers who train in such labs may go on to become lab heads, and continue the cycle of productivity without reproducibility.

Misleading or irreproducible research can have far-reaching consequences. Most published research that is preliminary in nature forms the basis of other hypotheses or research questions explored by others researchers with similar interests. If the initial hypothesis-generating research is not sound, and secondary publications expand upon, but do not attempt to validate the original findings, considerable time, money and effort is wasted, careers are affected, and real advances can suffer. In the worst-case scenario, wrong information may form the basis of translational work that enters clinical trials, exposing patients to potentially harmful treatments.  

Industry invests substantially in candidate drug targets sourced from published literature and conference presentations. The validity of candidate drug reports was highlighted by a team of Bayer researchers who retrospectively surveyed all sources of data that contributed to 4 years of in-house validation programs. They found that they were able to reproduce the relevant published findings for only 20–25% of the projects surveyed.

A similar survey by Amgen scientists found that of 53 clinical oncology publications that were deemed ‘landmark’ studies (21 were published by journals with impact factor >20), the findings of only 6 were corroborated by their in-house scientists. The authors of these reports discussed a range of explanations why validation attempts may have failed and the challenges of reproducing published findings, including variations in reagents and experimental models. 

A report by John Arrowsmith at Thomson Reuters on phase II projects from 16 companies, representing ~60% of global R&D spending, showed that success rates had fallen from 28% in 2006–2007, to 18% in 2008–2009. Analysis of 87 phase II failures from 2008–2010, with known reasons for failure, revealed that 51% was due to insufficient efficacy. Although historical trends show that efficacy remains the most common reason for failure, the proportion of efficacy failures decreased by 11% and safety failures decreased by 7% during the period 2013–2015.  

“To do better, insights on reproducibility will be crucial. Laboratory research is of tremendous importance. We should not drown its excellence in a sea of irreproducible results.” (John P.A. Ioannidis 2017)

These reports confirmed the general belief that there was an urgent need for all stakeholders, whether academic institutions, journal reviewers, or funding agencies, to adopt more stringent policies and implement initiatives to improve reproducibility. Basic research has for many decades been the foundation of discoveries that have led to vaccines, new drug developments, and a better understanding of disease processes that contribute to improved health.

Choosing careers in research is indeed a ‘hard slog’ and involves many challenges, and while the thrill of discovery may initially be the driving force, improvements in human health should remain the central focus of health-related research programs.

Despite the current concerns, it is not all ‘doom and gloom’ as data on 2013-2015 phase II & III trials showed that late-phase failure rates are declining and rates of progression at the regulatory review stage had increased. Also recent reports from the Reproducibility Project that aims to independently replicate high-profile research in cancer biology, shows promise.

In our next post we will review some of the methods that can be adopted to improve reproducibility and validation of research findings.  

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.

← Back to All Articles

Unfog the science…ensure quality, clarity and accuracy.