Blog

The current climate for research funding in academia, coupled with unmet public health needs and the low rate of return on research investment by industry, highlight the need for academic-industry partnerships, and a shift in what for decades was thought of as a ‘clash of cultures’.

Academic science is driven by education, intellectual curiosity and discovery, while the mission of industry is translational research, commercialization and profit. Academic scientists share their findings with the wider scientific community, while industry for obvious reasons tends to be reticent about this. The wall separating these cultures was first breached by university engineering and computer sciences sectors, leading to patenting, licencing and royalty income by major research universities.

A similar process has been occurring in the last 2–3 decades in the biomedical sciences, with adoption by many leading universities of a “20% rule”, encouraging faculty to engage in extramural activities with industry one day a week. Although such alliances have been viewed as productive overall, concerns continue to abound, including conflicts of interest, loss of public trust, increased workload for institutional review boards, and lower than expected financial return from licencing agreements and royalties.

Academics who are driven by ‘pure motives’ are currently facing decreased government funding for research and pressures to link up with industry. In commenting on innovation in Australia at a press conference in 2015 at Miraikan, Japan’s National Museum of Emerging Science and Innovation, Prime Minister Malcolm Turnbull said: 

“…we’re adding another criterion for success in achieving grants which is to demonstrate the degree of collaboration that you’re undertaking with business, with industry, so you can add “publish or perish” or perhaps “collaborate or crumble” as well.”

Recognizing the value of partnerships between industry and academia is not a one-way street. High quality research tracked by the Nature Index show that almost 90% of articles authored by corporate institutions are in collaboration with academic institutions, while only 2% of high-quality research come from the corporate sector. Partnerships between corporate and academic institutions have more than doubled since 2012 when tracking by Nature Index began, and half of them are in the life sciences.

Collaborations between academia and industry that are undertaken to address complementary parts of a research question can improve the publication power of the program. For the corporate side of the partnership, it means attracting and retaining top-tier scientists, which ultimately adds to the credibility of the research. A recent example is the 2013 collaboration between the Walter and Eliza Hall Medical Research Institute in Australia, and Genentech in the US, that led to the development and approval of venetoclax, a potent new therapy for patients with chronic lymphocytic leukemia.

The post-genome period and the prospect of new therapies have also helped to foster academic-industry partnerships. In 2016 AstraZeneca launched an integrated genomics initiative based in Cambridge, UK. This collaboration with The Wellcome Trust Sanger Institute, and the Institute for Molecular Medicine in Finland will focus on discovery of new drug targets and biomarkers across Astrazeneca’s main therapy areas. Professor David Goldstein, renowned for his research on genetics of disease and pharmacogenetics, and the director of the Institute for Genomic Medicine at Columbia University, is leading Astrazeneca’s ambitious 10-year genomics initiative.

It is also no secret that many academics in top positions with successful careers have been poached by pharmaceutical giants due to the growing need for innovation. Many academics who made the switch did so, not because of the prospect of better remuneration, but because advances in technology and the emphasis on translational research have made it particularly attractive to partner with industry where developing new therapies is a ‘hard’ endpoint. It is also the difference between research that may eventually help people, versus developing therapies that directly benefit people.

Such moves are not for everyone, because for some academics the greater motivation may be preclinical research and the joys of discovery. For others who want to see their innovations brought to fruition, the attraction to industry also depends on whether the pharmaceutical company is a good fit, and academic scientists can continue to espouse their academic ethos. An example of this was Dr Mark Fishman, who was chief of cardiology at Massachusetts General Hospital prior to accepting the top job at the Novartis Institutes for BioMedical Research. Fishman made a point of recruiting academic researchers because he liked their way of thinking.

“When you’re in academia, you have to develop critical thinking, and your ability to survive depends on your speaking and writing well and defending clearly what it is you want to do. The entrepreneurial spirit and culture of survival in academia is quite relevant to getting things done [in business]. (Fishman, Novartis)

Following retirement, Fishman was succeeded in 2016 by Dr Jay Bradner, a talented and innovative physician-scientist from the Dana-Farber Cancer Institute and Harvard Medical School. Under their leadership, Novartis has forged many more alliances with academia, and reported 2017 as a landmark year for innovation.

As with most collaborations and partnerships, academic-industry partnerships can work if there is an alignment of common goals, free and open communication, and trust between the parties in sharing ideas and data. The common ground between the two parties is identifying ‘druggable’ candidates in pre-clinical research that leads to the development of novel therapies. The divergence of academia and industry arises mainly in their respective approaches to achieving this endpoint.

Although many academics report their research in publications or grants, when it comes to therapeutic potential, their focus tends to be mainly on understanding a previously unknown biological mechanism or the role of a molecule in the disease process. Industry scientists or ‘drug hunters’ however, tend to be more focused on whether a specific molecule involved in the disease pathway can be effectively targeted for therapeutic purposes. Academic scientists may also be concerned about intellectual property and the risk of revealing promising results prior to publication for fear of ideas being scooped. There are also well-founded concerns that industry involvement may mean delayed publication and a tendency that they will ‘drop the ball’ in the face of negative results.

At a recent “Bridging the Innovation Gap between Academia and Industry” event held in New York, panelists Dr Barbara Dalton, Director of the Pfizer Ventures Investments team, and Teri Willey, Vice President of Business Development at Cold Spring Harbor Laboratory, provided some insights into traits needed to build a successful partnership:

1. Speak and write for the general population. A common complaint about scientists is that they do not explain science in a way that helps the general population understand what they are doing. The litmus test is whether you can tell your mother or grandmother—assuming they are not also scientists—what you are doing.
2. Good salesmanship. Scientists may have great ideas and may be doing ‘cutting-edge’ science, but they need to be able to sell their project. That means not only speaking and writing clearly, but developing a good ‘sales pitch’ and being able to persuade the funding organization or industry partner that they are on the right track with their project without crossing the line into ‘used-car salesman’ territory.
3. Leadership skills. Some scientists are more suited to leadership than others and can provide persuasive arguments in support of commercialization of project. Such individuals should be identified at the institutional level and supported as ‘business agents’ for liaising with industry partners, while maintaining their role in supporting the researcher whose work they are promoting.
4. Mutual appreciation of value. This involves striking a balance on more than just the financial aspects of the partnership. ‘Value’ varies with each entity, and can be one of the most difficult aspects of negotiating a partnership, but it is critical to success.

Dr Dalton also recommended investing in ‘entrepreneurs in residence’—people who have had experience with commercialization and can facilitate transitioning projects from the laboratory to a commercial sponsor.

Life sciences companies have fallen behind the technology sector and are facing lost opportunities by their reticence to share knowledge. The SEMATECH story, a research consortium formed in the mid 1980’s of fourteen computer chip makers in the US semiconductor industry, has become a model of how open innovation and drawing on the knowledge of your peers can transform an industry. Facing tough competition from Japan, this alliance banded together to solve shared problems and establish standards for chip design and manufacturing. Today US producers have not only regained their footing, they control half of the global semiconductor market.

Pharmaceutical companies have increased research spending by as much as 287% in the last two decades, and are facing lower revenues from expiring drug patents and longer time frames to bringing new therapies to market. At the same time there are many disease conditions with growing healthcare needs.

Although there are some very successful industry-academia partnerships, there is still a significant need for cooperation between academia and industry. One such effort is the Academic Drug Discovery Consortium which began in 2012. This collaborative network of academic and industry scientists, philanthropic and government organizations across 16 countries aims to share technologies related to drug discovery, and provide a platform for engaging with the life sciences industry.

If successful, academic-industry collaborations represent a move in the right direction, and may be the light at the end of the shrinking drug pipeline that could lead to improved translational research, greater innovation, and the much needed development of new therapies.

At SugarApple Communications, we support ethical scientific endeavours in academia, industry and government sectors. Let us help you find the best way to communicate your research. Get in touch today and let’s talk.  

Research partnerships between industry and academic institutions have more than doubled in the last five years, 50% of which are in the life sciences. This significant shift to seeking partnerships with academic scientists is evidenced by the fact that almost 90% of publications authored by corporate researchers are in collaboration with academic or government labs.

While corporations are known to protect their property, they see the benefit of such collaborations in producing high-quality research. The value of such alliances is also apparent because the public gives much weight to academic research as unbiased independent confirmation. Corporations should not fear associations with academics who themselves are fearless in defending their research, and should legitimately want to have such endorsements to back their products.  

Given the current emphasis in academia on translational research, it is more critical than ever before that ethics and integrity be in the forefront of all scientific endeavours.

Ethics and integrity in science are issues that have been in the spotlight recently, and are gaining momentum in many discussion forums and leading journals. The pursuit of truth above all else seems to be a fading tenet, as the need to increase publication quotas is tenuously balanced with career progression for many who love research. As reports of scientific fraud increase, and social media opinions on medical topics that are not based on good science flourish, it is becoming harder for scientists to retain their credibility and engage with the public. It is therefore urgent that the scientific community upholds the highest standards of ethics and integrity.

This issue has led to the development of a Code of Ethics by the World Economic Forum Young Scientists Community — a group of researchers under the age of 40 from diverse fields and from around the world. Their reflections and consultations with other researchers and ethicists identified seven principles that form the basis of the code that stakeholders are invited to endorse, with the aim of shaping ethical behavior in scientists and facilitating a cultural shift in scientific institutions. This code ranges from fundamental social behaviors such as ‘Minimize harm’ — as in damage to public health or waste of research dollars — to issues that are potentially contentious and subject to interpretation, such as ‘Pursue the truth’. Overall the code has stimulated much discussion, and serves as a timely reminder of the social and moral obligations of those who conduct research.

There are striking examples of scientists throughout history with a strong commitment to integrity and social responsibility. Rachel Louise Carson (1907-1964), is renowned for her work on pesticides and her campaign to alert the public of the harmful effects of DDT. Although she faced strong opposition from the chemical manufacturers, she urged awareness of the environmental impact of this chemical and advocated responsible use. Her work led to a grassroots environmental movement, the creation of the Environmental Protection Agency in 1970, and the banning of DDT use in the US six years after her death. Herbert Needleman, a child psychiatrist, is well known for his research on lead poisoning, and provided the first evidence that exposure to lead was associated with cognitive defects in children. His work was instrumental in the EPA mandate to remove lead from gasoline and interior paints.

Research integrity and social obligations are often based on individual value judgments. A researcher may decide to work in academia vs the commercial sector out of altruism and a desire to choose the research she wishes to pursue. For example a researcher working for Acme Pharmaceuticals may find that her research funding is determined by the potential for market share and financial benefit, while government and public funding sources may offer more flexibility in her choice of research topic, as well as autonomy in the discharge of social and moral obligations to the public. In either scenario, personal values come into play, which may be challenged by the interpretation and conclusions drawn from their research.

Ethics and integrity, loosely defined, is an intrinsic value system that governs the ability to distinguish between right and wrong, or acceptable and unacceptable behaviour. Most people consider these as matters of morality or simple common sense. Given the range of disagreement on what constitutes ethical norms, the ‘common sense’ approach may in fact be overly simplistic, and depends instead on the context. Ethics in the medical sciences may have more direct and obvious application at the individual level as defined by the Hippocratic Oath “First, do no harm”, while issues like global warming tend to be more complex, requiring a broad range of stakeholders, e.g. politicians, economists and environmental scientists to work together to achieve an ethical outcome. In the latter scenario, each stakeholder would have a different set of moral and ethical responsibilities that makes it difficult to achieve a common goal.

Research institutions almost universally have policies and training modules in place that outline ethical standards for personal behaviour (e.g. workplace bullying and sexual harassment) and research conduct (e.g. animal ethics, plagiarism and fabrication of research data), and strict requirements for timely completion of these modules.

There is also much emphasis internationally on excellence in science, a recent example being the launch of a Regional Centre for Research Excellence at the University of the West Indies last month. Defining and measuring excellence is difficult and engenders much debate, particularly because of a lack of consensus across different disciplines. While there is disagreement on what constitutes excellence in research, “soundness” and “proper practice” that maintains ethics and scientific integrity remain a common thread across disciplines in the pursuit of excellence.

Regardless of cross-disciplinary variations, ethics and integrity in science is important because it promotes:

• A quest for knowledge, truth, and minimizing error. This includes prohibitions against falsifying or misrepresenting research findings.
• Collaboration and cooperation between institutions. The success of collaborative work is highly dependent on common ethical standards, and promotes trust, respect, and responsibility, as exemplified by the ICMJE guidelines for authorship.
• Accountability in adhering to policies on the protection of human subjects, animal care and use, and declarations of conflicts of interest particularly for publicly funded research.
• Public support and trust. While government resources vary according to economic and political interests and agendas, research funding from philanthropic organisations is a growing phenomenon worldwide.

The public relies on the integrity of scientists who in turn can earn their trust through a wide range of avenues, from providing expert testimony highlighting the legal and policy implications of research, to whistle-blowing on scientific misconduct, to advocating for public health policies. These avenues can be uncomfortable for many scientists by bringing them sharply into the public spotlight. Like Carson and Needleman, whose work led to major public policy shifts, the dilemma that scientists with integrity may face is criticism and backlash from opponents and industry groups who may have more to lose from these policy changes.

Overall, scientific integrity can be summarized in many ways, but simply stated, it is founded upon:

• honesty in reporting data and results
• objectivity in data analysis and interpretation
• carefulness and record keeping of research activities
• openness in collaboration
• protecting patient confidentiality
• respect and acknowledgement of all contributors, i.e. giving credit where credit is due
• accountability and responsible publishing, and
• communicating values through mentoring.

A model example of scientific integrity at its best occurred when a paper with significant clinical impact published in 2014 by the Journal of the American Medical Association, was discovered two years later to contain analysis errors because of miscoded data. The error turned the original findings on its head, spelling doom for the original published paper. In what has been highlighted as “a shining example of scientific integrity”, the lead author acted quickly to submit a ‘retract and replace’ article. JAMA agreed, using the opportunity to urge authors to share data and avoid the stigma of retraction when honest errors are discovered.

“Think of the unthinkable when it comes to checking the quality of your data; If you think that all possible has been checked, check again; Always let others use your original data for new (or just the same) analyses.” (Marc Bonten 2017 Blog post)

This is in stark contrast to a situation where for example a freelancer commissioned by Acme Pharmaceuticals discovers errors in her data while drafting a manuscript, and assumes that data checks prior to write up were not part of her responsibilities. As outlined in my recent article on authorship, ethical conduct in reporting requires that all parties involved in the preparation of a manuscript for publication conduct multiple data checks and take responsibility for the contents of the published manuscript.

Correcting major errors prior to publication, even at the stage of checking the pre-publication proofs, may be a lengthy process if all authors need to approve these corrections, but it should be done even if it delays publication. The decision to correct the error should be made by all authors if the correction substantively changes the take-home message of the paper.

Reputable journals and scientists with ethical standards can work together to preserve the integrity of their research by correcting any errors as soon as they are discovered, and simply get on with the business of science.

At SugarApple Communications, our mission is to adhere to the highest ethical standards in the promotion of high quality research. Let us help you find the best way to reach your intended audience, and assist with writing, editing and statistics. Get in touch today and let’s talk.

The UN General Assembly declared 11^th February as the International Day of Women and Girls in Science, aiming to achieve full and equal access to participation in science for women and girls. This is vital for achieving the 2030 Sustainable Development Goals adopted by world leaders in September 2015. A recent survey conducted in 14 countries found that compared to men, women are half as likely to graduate with a Bachelor’s or Master’s degree, and one-third as likely to graduate with a doctorate in a science-related field.

“We need to encourage and support girls and women achieve their full potential as scientific researchers and innovators” (UN Secretary General Antonio Guterres)

Although a wide range of factors influence the careers of women in science, perhaps disproportionately in comparison to men, this article aims to highlight some of the achievements made by women in science, and some personal observations during the course of my career in science. It also highlights principles that apply equally to men and women, and the fact that anyone can work to their best potential if they commit to working with integrity and honesty, and observing a high level of ethical conduct in their careers.

As someone who loved the biological sciences since early high school, and who grew up in a household where male and female siblings were in equal proportion, I was blessed to have parents who were education professionals each in their own right, and who encouraged (in fact, expected) us all to excel in our respective fields and professional pursuits. Gender bias may have existed, but it did not affect the way I saw myself or others in terms of career goals or simply being the best I could be. As expressed by Prof Michelle Simmons in her recent acceptance speech on receiving the 2018 Australian of the Year award, believing what others think of you can become a self-fulfilling prophecy.

Significant contributions by women to science and medicine go back centuries. Although women were excluded from university education in the earliest emergence of universities, some countries were more liberal than others. The first woman to chair a scientific field was Laura Maria Caterina Bassi, a physicist and academic, in the late 18^th century in Italy. Segregated women’s colleges arose in the 19^th century, and in 1903 Marie Curie was the first woman to receive a Nobel Prize, which was in physics. She then went on to receive another in 1911 for chemistry, and her work on radiation is well renowned.

Although a total of forty women have been awarded the Nobel Prize by 2010, not all women can or wish to pursue this level of achievement. Many women are scientists because they love it, and simply enjoy discovery and want to do great science. Prof Michelle Simmons rightly stated “women think differently, and that diversity of thought is invaluable to technological research and development”. We all know that scientific research needs to be approached from many different angles, and this can only enhance the findings and lend credibility to research outcomes as a whole. Prof Simmons also did not believe in “mandating equal numbers of men and women in every job”. It is just as wrong to hire on the basis of gender as any other characteristic, unless that characteristic helps to advance the field. Girls and boys in early education should have the same opportunity to develop their potential in STEM disciplines without bias or stereotyping. Their differences will shine in positive ways if they are encouraged to develop their talents, whatever that may be.

I have been in science for 35 years, and all except one of my supervisors were women, although this was not my objective. This is not counting my PhD advisory committee who were all men, and whom I chose because they were the best mentors for me. One in particular, a statistics professor who also served on the FDA CardioRenal Advisory Board, has remained someone I respect and admire because of his fierce and unwavering ethical stance on many issues, particularly the interpretation of clinical trial data. When I approached him, he surveyed me skeptically as yet another epidemiology student needing a statistics professor on her committee. By the end of the interview, he was completely disarmed in his attitude towards me, and from that point on it was evident that we had struck common ground in our views on ethics in science. The earned respect was mutual, and being an ‘older’ PhD student at the time, I was not intimidated by his occasional ‘wrath’ over issues of scientific misconduct and poor data interpretation.

But I digressed! Getting back to the women who were my supervisors, I was fortunate to have had many who were excellent role models and mentors over the years. Some of the most outstanding qualities I have observed in successful women in science is the ability to be organized, to multi-task, and to lead by example. Honesty and integrity, discipline, respect for employees’ personal lives, fairness, challenging junior scientists to excel and giving them space to develop their thinking skills while offering calm and constructive guidance, and giving credit where credit is due, are some of the characteristics I have valued most in the women in science who were my mentors.

I also often bring to mind a situation where a difficult decision needed to be made, and a senior female scientist who overheard me discussing this, came over and whispered to me ‘To thine own self be true’. This has remained with me over the years and continues to ring true in many situations. But it takes courage and conviction to apply this and requires weighing up the pros and cons of each individual circumstance where a decision has to be made.

There are certain principles that should govern all aspects of our life, whether in family life at home, or in our working environments, or in our relationships. The measure of the person is the consistency of this in all walks of life. There may be times when you are challenged to stand up for your principles, and doing so can cost you your standing on a committee, or your reputation. But can you sleep at night knowing you were not true to yourself or the research you love and for which you strive for excellence?

I have often thought of how best to explain ‘leadership by example’. The golden rule, ‘do to others as you would have them do unto you’ comes to mind. The sorts of people who issue edicts that they themselves either cannot or will not fulfill, are those who rarely gain the respect of their peers, at least not in science. Those who criticize others and demand what they themselves cannot deliver, while blinded to their own incompetence, do not make good leaders in any field.

It is also important for women in science to take care of their personal lives and pay attention to work-life balance. This has also recently received attention in a highly visible journal article. As a scientist you need to have clarity of thought. Few things can cloud the mental landscape and your research progress more than a personal life that is disorganized and in disarray. This takes work and discipline, as well as considerable support from family and close friends. It is not your sole responsibility as a woman to organize everyone, but like your supervisory role in science, delegating responsibility in a kind and supportive way, while not shunning it yourself, is critical to success.

As part of ethical conduct, keeping your word is also important to your success as a scientist. I have had colleagues who offer to do a job, and they not only never get it done, but their mind seems to go blank when it is brought to their attention, usually in an effort to think of an excuse. We all forget things at times, and to err is human. Taking responsibility for something you overlooked will enhance your credibility far more than making excuses. Eventually excuses will be embarrassingly in limited supply, and the cold hard facts of your unreliability will cost more than you realize. So don’t go there. Take ownership for both your achievements and your failures.

Some final characteristics I have valued in the women in science whom I’ve known personally, particularly here in Australia, are conviction, graciousness, humility, humanity, and a sense of humor. These will elicit loyalty and support, and the willingness of your staff to voluntarily go the extra mile like no pay rise will. I believe this is the icing on the cake that will go a long way in attracting the best and brightest scientists to your team.

Finally, there is so much more women in science can do to advance their respective goals, by simply being true to themselves and not trying to be what they are not.  Careers in scientific and medical research are universally known as a ‘hard slog’, and participation by women will continue to bring to science those qualities that enrich the tapestry of human endeavor.

At SugarApple Communications we celebrate all scientists and the labor of their research. We can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.

Feature Image AlesiaKan / Shutterstock.com

It may be obvious to many that an association between two factors does not necessarily imply causation. We are frequently exposed to scientific reports that factor X is associated with disease Y, which are then altered to ‘X plays a key role in Y’, and we are led to believe that X causes Y. We also find reports on social media of individual exposures that are believed to have caused disease simply because they happened at the same time. Such reports have led to false claims about novel associations and their relationship as a cause of disease.

For example, a positive relationship between the amount of damage caused by fires and the number of firemen at the scene does not mean that sending more firemen to a fire causes more damage. Also, high coffee consumption may be associated with a decreased risk of skin cancer, probably because high coffee consumption is associated with indoor lifestyles and activities, and therefore less exposure to the sun. But coffee itself does not protect against skin cancer, and is therefore not causally related. 

Food has been a prime candidate in the search for causes and cures for cancer as far back as the 18^th century. In an innovative study, Schoenfeld and Ioannidis addressed the question “Is everything we eat associated with cancer?” The authors selected 50 common and familiar ingredients from random recipes in a popular cookbook, and queried them on PubMed for an association with cancer risk. They found articles for 40 of these ingredients that the authors of the articles claimed was evidence that they either increased or decreased the risk of cancer. Some ingredients had effects on both sides of the risk profile, and had unconvincingly large effects that tended to shrink in meta-analyses.

So how do we decide whether an observed association is evidence for causation or not? Students of epidemiology or public health are taught to differentiate between association and causation, but may be tempted to exaggerate the implications of association studies when they enter the ‘publish or perish’ world of academic research. An inherent weakness of observational association studies is that experimental studies may not corroborate their findings. Inferring causation from a single association study may therefore be misleading, and could potentially cause harm to the public. This is a major reason why preliminary results from association studies should be interpreted with caution, and if publicized, should be carefully presented, keeping in mind the aims of the study and ‘real world’ implications as opposed to statistical significance.

“All scientific work is incomplete – whether it be observational or experimental. All scientific work is liable to be upset or modified by advancing knowledge. That does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time.” (Sir Austin Bradford Hill 1965)

In 1965, at a meeting of the Royal Society of Medicine, Sir Austin Bradford Hill outlined nine tenets to consider when deciding whether causation was a factor in an observed association. He clearly stated that he had “no wish, nor the skill, to embark upon a philosophical discussion of the meaning of ‘causation'”. The starting point in assessing a causal relationship is generally an observation of an association or correlation between an exposure and an outcome that may or may not be attributed to the play of chance. These tenets are as follows:

1. Strength of association. This refers to the magnitude of the effect of the exposure on the disease compared to the absence of the exposure, often called the effect size. This is represented by the odds ratio, confidence interval and p-value. These measures should be considered together when deciding how strong or how real is an association.
2. Consistency. The association remains even when other factors change, e.g. different time, place, location, ethnic groups, age groups, gender etc.
3. Specificity. The causal factor is quite specific to the outcome. For example if coal mine workers exposed to coal dust develop black lung disease, whereas those not exposed to coal dust do not, then coal dust specifically causes black lung disease. Working in a coal mine may not be the causal factor, as a person may be exposed to coal dust outside a coal mine. In an association study, it is important to isolate what is specific to the disease process to determine if the association is causal.
4. Temporality. This answers the question: Which came first? You would expect that if an exposure causes a disease, then the exposure should necessarily precede the disease development.
5. Dose-response or Gradient. Evidence that an exposure causes a disease may be related to a certain quantity or dose of the exposure, in which case you may see varying degrees of disease depending on the extent of the exposure.
6. Biological Plausibility. Plausibility asks the question: Could the observed results fit into an established biological theory if it existed? This helps argue for causation, but it is not absolutely necessary as the understanding of the disease biology may be immature.
7. Coherence. Assessment of causation should not conflict with existing knowledge of disease biology. Coherence asks the question: If the association is indeed causal, would it fit into an existing biological theory? The difference between ‘plausibility’ and ‘coherence’ is subtle. Coherence assumes there is an existing biological theory, and rejects the result if it does not fit into that theory, while plausibility at least allows for it in the absence of mature science.
8. Experimental evidence. This has also been called ‘challenge–dechallenge–rechallenge,’ meaning, if we prevent the exposure, is it likely to prevent disease, and if we re-introduce the exposure, does the risk of disease return? The best experimental evidence for causation comes from randomized controlled trials, although in some circumstances this may be unethical.
9. Analogy. Causation by analogy implies that if an exposure is known to cause disease, then it is highly likely that a similar exposure under similar circumstances will also cause disease.

Bradford Hill tenets are not meant to be a checklist for assessing causation, nor are they intended to be adhered to pedantically, but should serve as a guide when evaluating whether an exposure might be causally linked to a disease. It is unlikely that any single association study will satisfy all criteria for causation, but any given study may address some of the nine tenets, or none at all.

In fact, inferring causality may not require association studies or even a significant p-value. A classic example of this was the drug thalidomide that was approved in Europe in 1957 for combating morning sickness among pregnant women. Subsequently, an explosion in the incidence of neonatal deaths and congenital birth defects, of a type that can only be described as horrific and extremely rare, occurred almost simultaneously in 46 countries where this drug was approved¹. Clearly, any study attempting to associate thalidomide with birth defects would be unethical. Also, the extremely low prevalence of this type of birth defect in the general population, coupled with the striking increase in its prevalence in countries where thalidomide was prescribed, require no statistical measure of association to infer causation.

Bradford Hill tenets are not irrelevant or outdated, and provide useful principles for establishing causation. With new technologies and advances, various scientific disciplines may contribute to a better overall understanding of the disease process that can enhance the application of these criteria, and provide a stronger argument for or against causation.

In the absence of indisputable and compelling evidence that an association is causal, it is important to consider the entire body of knowledge and think through all sources of evidence when determining whether an exposure causes an outcome or disease. As a well-respected statistics professor of mine frequently reminded us, causality is a ‘thinking person’s business’, i.e. don’t let your computer or statistics program, or for that matter, anecdotal or biased reports, decide on the evidence.

As researchers we experience a eureka moment when the output of our statistics program generates a p-value with many zeros after the decimal point, i.e. a negligible probability that our finding was due to chance, but become crest-fallen when a validation study generates a p-value suggesting that our earlier result was well and truly a chance finding. This does not mean that the study should be filed away in the endless repository of unpublished science, nor should it be spun into something it is not. Negative results, assuming they were based on sound methodology, are not failed research, but are an important part of ultimately assessing causality and obtaining definitive answers to research questions, as highlighted in my previous blog on systematic reviews.

In a future article I will go into more detail on some of the common statistical measures that are reported in the scientific literature and their implications for ‘real world’ evidence.

¹More on the Thalidomide tragedy can be found in the book ‘Suffer the Children: The Story of Thalidomide’ that chronicles this disaster.

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.

In this article, we focus on some of the key elements to look for in a systematic review, and how to assess its credibility, when searching for answers to a specific clinical question. As outlined in our last blog, a systematic review is a summary of all relevant studies on a clearly formulated clinical question, using systematic methods according to a strict, pre-defined protocol. It often involves meta-analysis, a statistical technique for pooling the results from different studies to provide a single estimate of effect. A major limitation of systematic reviews is that they are only as good as the studies they summarize.

For simplicity, we define ‘treatment’ as any intervention, exposure, or clinical attribute that is being assessed in the systematic review, and ‘placebo’ as the comparison intervention or exposure. 

1. There should be a clear focused clinical question.

As with any individual study report, authors should clearly state the clinical question. This should include four elements, often referred to in the literature as PICO; the patient (P) or study population characteristics, the intervention (I) or exposure or treatment regime, a comparison (C) intervention or treatment, and specific outcomes (O).

2. Is there sufficient detail on how the literature search was conducted?

A well designed systematic review should involve details of how studies were identified, e.g. what electronic databases were used to retrieve studies, language restrictions, and any additional sources of data including clinical trials registers, conference reports, and whether any unpublished studies were included.

If the search for relevant studies is not exhaustive, the results of the systematic review may be flawed. For example, one study showed that searching only MEDLINE retrieved 55% of eligible clinical trials. It is important to use multiple electronic databases including EMBASE and The Cochrane Library, using various search terms, medical subject headings (MeSH) and synonyms to yield the best results.

3. Are there pre-defined criteria for which study types will be included?

A systematic review that involves a therapeutic intervention, or will contribute to clinical guideline development, should prioritize randomized controlled trials where available, as these are more reliable and less subject to selection bias compared to observational study designs. Systematic reviews that aim to assess the adverse effects of treatment may include observational studies, such as case-control studies and post-marketing surveillance studies.

PRISMA guidelines recommend that more than one contributor should be involved in selecting and reviewing the studies for inclusion, in order to avoid subjective decisions. The kappa statistic (κ) of inter-reviewer agreement should be estimated and reported to provide readers and those using the results with a degree of confidence in the systematic review.

4. Does the systematic review include meta-analysis?

Meta-analysis is a statistical technique that combines the results of multiple studies to produce a single estimate of effect, which tends to be more reliable than those from the individual studies because it is based on a larger sample size. However, meta-analysis should only be performed if the individual studies are sufficiently similar in terms of the PICO (patients, intervention, comparisons and outcomes). It is therefore important that the study question has a relatively narrow focus. For example, consider the following questions:

A. What is the effect of all cancer treatments on cancer outcomes?

B. What is the effect of chemotherapy on ovarian cancer survival?

C. What is the effect of carboplatin-based chemotherapy on ovarian cancer-specific survival?

D. What is the effect of standard doses of paclitaxel and carboplatin chemotherapy on ovarian cancer-specific survival?

Question D considerably narrows the focus of the overall research question to a specific treatment for a specific disease condition and a specific outcome measure, and is more likely than Question A to provide a meaningful result with clinical application. However, the results of Question D will need to be carefully applied, as the question does not address differences in population and other aspects of the disease biology.

5. The meta-analysis should include a test for study heterogeneity and the results interpreted.

Meta-analysis should always include a statistical test for heterogeneity. This assesses the consistency of the results or variation in outcomes, across included studies. Most tests for study heterogeneity generate a p-value that should be reported and interpreted by the author in the context of the clinical implications of the study findings. One of the best measurements of heterogeneity is the I² statistic which describes the proportion of variation across studies that is due to heterogeneity rather than chance. In real life, study heterogeneity may mean that the treatment effect may differ between patient groups, possibly according to ethnicity, age, gender etc.

6. The results of meta-analyses should be graphically displayed with a forest plot.

Forest plots are the most effective way to present individual estimates from the input studies included in the meta-analysis, as well as the single summary estimate derived from the meta-analysis. Study estimates are most commonly expressed as an odds ratio comparing the treatment vs. placebo or comparison intervention, along with their 95% confidence intervals.

The odds ratio is simply a ratio of the effect of the treatment to that of the placebo. It is often called the effect size, and is derived from the simple division of the size of the treatment effect over that of the placebo. If both are very close, then the result of this division is 1, also known as the ‘null’ value, i.e. no difference in outcome between the treatment and the placebo.

The confidence interval is a range of likely effect sizes for the study population and contains the true estimate of effect. It also indicates how ‘confident’ we can be in the results. Narrower confidence intervals indicate that the effect size is very precise or believable, and close to the true population effect, whereas a wide confidence interval suggests that the effect size is very variable and imprecise, is less believable, and should be interpreted with caution.

The vertical axis of the forest plot represents the ‘null’ value of no difference between treatment groups. The odds ratios for individual input studies shown in the forest plot is often depicted as a square, the size of which depends on the sample size of the study, and therefore the ‘weight’ it carries in the meta-analysis; the line drawn through this square represents the confidence interval. The summary estimate from the meta-analysis is typically a weighted average of the results of individual input studies and is often represented on the forest plot as a diamond shape closest to the horizontal axis. The vertical points of the diamond indicate the summary effect estimate, and horizontal points indicate the range of the confidence interval.

7. The systematic review report should include commentary on bias or study limitations.

A well-conducted systematic review should report information that will help readers decide on the applicability of the results. It should include some commentary on possible sources of bias, e.g. publication bias arising from the tendency of journals to publish studies that have positive effects. Language bias can also be a factor if studies are selected because they are published in an English language journal. Authors of systematic reviews should comment on the risk of bias in the individual studies included, and their interpretation of the result of meta-analysis in the context of these limitations.  They should also include an explanation of study heterogeneity as a potential limitation, as outlined in point #5.

8. An interpretation of the results and implications for clinical practice or further research should be provided.

The authors should provide an interpretation and explanation of all reported statistical estimates and their meaning in terms of the clinical application of the study findings. Readers of systematic reviews can also visually inspect forest plots to identify differences in effect estimates, overlapping confidence intervals, and the direction of the effect from individual studies, i.e. are most of the odds ratios or squares in the forest plot falling on one side of the ‘null’ line or both sides?

Effect estimates falling on the left of the ‘null’ line indicate that the treatment has a favorable effect on the outcome compared to the placebo; those falling on the right side of the ‘null’ line suggest that the treatment has a worse effect on the outcome compared to the placebo.

A similar judgment can be made for the summary estimate from a meta-analysis to gauge how confident you can be in these results. However, it is incumbent on the authors to provide details of their interpretation, implications for clinical practice, or whether the results are ready for clinical application, and what limitations to its application should be considered.

There is much debate in the scientific literature that systematic reviews cause research waste in light of the mass production of such publications that are poorly designed and conducted, with exaggerated claims. Judging the quality of systematic reviews is a first step in determining how credible the findings are, whether the methods conform to PRISMA guidelines for conduct and reporting, and how confident we can be in applying their results to healthcare.

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.

Healthcare research is becoming increasingly more accessible as the level of reporting and commentary rises, both via established news sources and on health-related internet sites and private blogs. The public appetite for health information is growing as evidenced by the popularity and uptake of such information, regardless of whether or not it comes from a trustworthy source. A well-written narrative review, or a carefully designed systematic review and meta-analysis, is important to summarizing healthcare research. 

We are bombarded almost daily with new and often changing reports about the latest research — “drink red wine”…“don’t drink red wine”…“drink red wine once or twice per week”… and so on. As we discussed in an earlier blog, overly ambitious claims may be contributing to a general mistrust, within society, of medical advice and guidance, even when the supporting data is reliable.

A recent commentary in Nature suggested that more collaboration was needed between computational experts and population-health researchers in order to synthesize health evidence as more and more data become available. Reviews that aim to summarize the available research to provide the ‘bottom line’ are a valuable resource for clinicians, researchers and the community as a whole. Such reviews can be conducted in a number of ways – from narrative reviews or opinionated summaries to more analytical, protocol-defined systematic reviews capable of providing summary estimates that can guide health policy.

Narrative reviews are traditional literature reviews that describe and summarize current knowledge on a specific topic or area of research. Authors of published narrative reviews should ideally have spent their careers working in that field. They generally use informal and subjective methods to screen and select studies to include, and apply their expertise on the topic to interpret and summarize these studies to provide a snapshot of the current research. Narrative reviews are largely qualitative, and tend to rationalize the diversity of information around the topic into a single coherent article, which critically evaluates the research, highlights gaps in the existing knowledge, and may even recommend areas that need more work. Authors should ideally also include a summary of the search criteria utilized, the inclusion or exclusion criteria, and their rationale for selecting them.

Narrative reviews may be voluntary, or commissioned by journals from leading researchers, on topics of current interest, with the purpose of obtaining consensus statements and perspectives on where the current research lies, and also to evaluate trends in research to direct future publications.

Systematic reviews, in contrast, aim to identify, evaluate and summarize all relevant studies on a clearly formulated question, using systematic methods according to a strict, pre-defined protocol. A systematic review should combine the results of selected individual studies to obtain a single more reliable overall estimate. The methodology involves a rigorous approach to ensure all possible, relevant research is considered, thereby minimizing the risk of bias. The criteria for inclusion should be transparent, and reasons for inclusion and exclusion clearly stated. All methods including search terms, criteria for selection, appraisal of the studies, and statistical methods, should be reported to allow study replication if needed.

The trustworthiness of a systematic review depends on a priori planning (both with regard to study selection and analytic approaches), careful documentation of a planned protocol agreed upon by those involved, and strict adherence to the set protocol. This avoids subjective or arbitrary decisions with respect to data extraction, or bias and selective reporting of findings, which could negatively impact healthcare decision-making.

Authors of systematic reviews should include commentary on the quality of each input study, an interpretation of their summary result and how best it can be used in clinical practice, their confidence in the result, and how they differ from those of other published studies. The old adage ‘rubbish in – rubbish out’ holds true for systematic reviews, and careful study selection is pivotal to their utility.

“Meta-analysis is the analysis of analyses … the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” (Glass 1976)

Meta-analysis is the statistical technique used to combine and summarize the results of quantitative studies to provide a single more precise estimate. It is a common misconception that the terms ‘systematic review’ and ‘meta-analysis’ are synonymous. They aren’t. Some systematic reviews involve studies with results that cannot be pooled statistically. It would, therefore, be inappropriate to apply meta-analysis to such studies.

The use of meta-analysis is not limited only to systematic reviews. Meta-analysis can be applied to small studies that address similar questions, and is particularly valuable where the individual estimates are inconclusive due to the small sample size. Combining these studies in a meta-analysis can improve estimates by increasing the sample size and therefore the statistical power to detect a true effect.

Critics of meta-analysis argue that this technique amounts to combining ‘apples and oranges’ that will generate an essentially meaningless estimate. These are valid concerns for small observational studies that have considerable study heterogeneity, or where there is variation between the results of individual studies. Ideally, meta-analysis should only be applied to studies with sufficiently similar design, study population, intervention or comparison, and outcome measures. The extent of variation or heterogeneity between studies can be assessed using a number of available techniques.  Authors of a systematic review should report the p-value for formal statistical tests for study heterogeneity, and provide some discussion on this in the context of the overall utility of their findings.

Much work has been done to standardize protocols for the conduct and reporting of systematic reviews and meta-analyses. The most widely endorsed and accepted is PRISMA, which was published in 2009. These and other guidelines on reporting health research studies can be found on the EQUATOR Network, an umbrella organization of multiple stakeholders with the goal of improving the quality of research and research publications. The Cochrane Library and Campbell Collaboration regularly publish systematic reviews and their protocols. Affiliated with Cochrane and EQUATOR is PROSPERO, an international database that prospectively registers systematic review protocols, which was launched through the University of York (UK) in 2011. By 2016, the number of prospective registrations reached 20,000 records from 107 countries.

Narrative reviews, systematic reviews and meta-analyses are all useful ways to evaluate and summarize large numbers of studies on the same subject. However, they are only as good as the studies they summarize, and the relative merits and limitations of each should be appreciated. This is critical when reviews are used to guide medical decisions and develop clinical guidance. Systematic reviews that follow clearly defined protocols and adhere to the guidelines for good conduct and reporting practices are a gold standard that provides the most reliable estimates.

In an upcoming article, we will discuss ways to identify characteristics of a sufficiently credible, well-written systematic review that ticks the boxes for good reporting practice.

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.

Reproducibility in research is the ability of a researcher to duplicate the results of another study with a degree of concordance that makes the original study at least believable. The impact of irreproducible results was discussed in my last blog.

Global investment in biomedical research is over US $100 billion annually, and leads to major breakthroughs that pay dividends to human health. But not all research translates into benefit, however noble the intention of the researcher. Many studies legitimately test valid research hypotheses that might shed light on disease processes, but may prove to be unactionable. It’s been estimated that ~85% of research investment is wasted annually because of problems that can be corrected. More information on reducing waste and increasing research value can be found in a Lancet series of five articles published in 2014.

Although deliberately falsifying or fabricating data is most damaging to science, it is less frequent compared to more widespread forms of misconduct that can often fly under the radar. The ‘most worrying misbehaviors’ cited in a survey of ~1300 scientists attending international research integrity conferences, were protocol deviations, selective reporting of positive results and insufficient reporting of study flaws and limitations, quality assurance failures, poor mentoring of junior scientists, and turning a blind eye to the misconduct of co-workers.

While there are many drivers of the scientific culture that generates ‘sloppy science’, it is quite possible to remedy this by introducing practices that have worked for some scientific disciplines. For example, guidelines developed by the NCI-NHGRI Working Group on replication in genetic association studies have led to large-scale collaborations that have transformed genetic and molecular epidemiology from one of spurious associations, to a highly credible one.

Standards exist in many industries that define best practice for various systems that ensure consistency, quality and integrity, and procedures to ensure compliance.

The following list is by no means exhaustive, but outlines some of the ways to ensure reproducibility.

• Schedule routine and informal ‘catch-ups’ between lab personnel. If done in the spirit of collaboration and good scientific citizenship, this will go a long way to encourage honesty and admission of errors without fear of retribution. A major failure by lab heads is a ‘hands-off’ approach to problems that foster poor morale that could lead to ‘sloppy science’.

• Develop a system of sample labeling, logging and storage organization. The aim should be about accuracy rather than convenience or quick retrieval. Where multiple users access the same samples, a log should be maintained of where the sample came from, how old it is, and who did what to it. An open and transparent lab culture, good citizenship, and collaboration and cooperation go a long way to ensuring quality research.

• Standardize protocols. The old adage ‘if it ain’t broke don’t fix it’ has no better place than in lab-based research. Protocols should be standardized and followed to the letter. If there are deviations, they should be documented and reasons given. In the interest of reproducibility, it should be standard practice that another individual in the group repeats the experiment. Senior research staff with many years of valuable experience and ‘magical hands’ should be encouraged to do the final replication before publication. 

• Don’t skimp on equipment maintenance. Equipment and key lab instruments such as pipettors should be calibrated on a regular schedule and documentation of this kept.

• Good technique is everything. Correct use of instruments should be an integral part of training for junior lab members. I’ve had the experience where I could not understand why one person’s 50 μl volume was consistently ~10-15 μl more than it should be. It turned out he was a bit ‘heavy-handed’ with his pipettor. The simple ‘eye-ball’ test by experienced lab personnel will quickly identify the source of problems that contribute to variability and experimental errors. 

• Get rid of old and outdated reagents. Most research institutes and organizations have clear policies on this. Where there are budget constraints it can be tempting to push ‘old’ reagents beyond their ‘best by’ date to save money. This can cost more in the long term. 

• Lab records and notebooks should be carefully and correctly filled out. These are legal documents for most institutions and cannot be removed from the lab. Electronic lab notebooks are on the rise, and allow protocols and methods to be aligned with the final published data where institutional audits are necessary. While random audits and spot-checks are commonplace in industry, this has yet to be widely implemented in academic institutions.

• Report methods in detail. Where journals have a word count requirement, additional details of protocols and methods can be included in Supplementary Information, which most journals have no word count limitations on. This will go a long way to others replicating your experiments.

• Publish negative findings. A considerable amount of blame for the bias in scientific literature can be laid at the feet of journals that preference positive findings. It should also be the responsibility of investigators to adequately address unexpected negative findings, without trying to put a positive spin on them. A respondent of a recent survey by Nature reported that he expected rejection of a manuscript outlining why a technique had failed, and suggests that the reviewers accepted his paper because he offered a solution to the problem. One of the best papers I wrote was a replication study using well-curated data from the largest multi-centre study to date, and very robust analysis. We made good use of the Supplementary Information to detail all methods and results. Our findings refuted other positive claims, and an Editorial by the journal confirmed that the study was sufficiently robust to conclude that our negative findings settled the question of clinical relevance.

• Pre-registration of a priori hypotheses. This has been one of the most publicized recommendations to improving reproducibility. It involves submitting an a priori hypothesis and analysis approach to a third party, before undertaking experiments, to guard against the temptation to pick potentially false positives that were not part of the pre-specified action plan.

• Robust study design and analysis approaches. Much has been said about this, and it continues to be one of the most vexing issues in both lab and computational research. Approximately 90% of respondents to the Nature survey ranked “more robust experimental design”, including blinding and experimental controls where possible, and “better statistics” higher than institutional incentives for improving reproducibility.

Scientific integrity and good practice in academic research is generally assumed to be the responsibility of individual lab heads and lead investigators whose career or funding is potentially at stake. Ultimately, their failure can tarnish the institute’s reputation, as funding bodies increasingly publicize institutional-based summaries of overall funding and achievements. Good scientific practice should therefore be cultured at the institutional level, and a system of guidance and compliance can be developed and formalized both within and across institutes.  Ultimately there needs to be a complete shift in culture by all stakeholders including investigators, institutions, funding bodies and journals, of rewarding best practice. 

“Academic metrics need to be devised that distinguish citations of discredited claims so that it is not more advantageous to state and retract a result than to make a solid discovery.” (Jan Conrad, Nature Comment 2015) 

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.

The issue of irreproducible research has been of concern to scientists for decades, but recently there has been considerable focus on this problem by leading journals in various commentaries and editorials, and calls for recognition and response to this problem by the scientific community.

Pressure to publish has dominated academic research for over half a century, and policies that inadvertently reward quantity at the expense of quality, and the rising focus on citation counts and other metrics that are increasingly used as a proxy for impact, have contributed to concerns that we are drowning in false or exaggerated claims. Science publishing has mushroomed into a global industry with new players aiming for market share in a climate that values publication numbers as a marker of success and a currency for academic advancement.

The exponential growth in numbers of publications and higher numbers of citations of articles, regardless of the quality has led to the belief that we are in a highly productive era. A recent report of the most influential biomedical researchers identified more than 15 million authors of more than 25 million scientific papers published in between 1996–2011. Analysis of publication patterns of over 40,000 researchers who published two or more papers in the first fifteen years or their ‘early-career’ phase, found that research productivity had not increased for most disciplines, taking into account co-authorship.

It is not an easy decision to know when to publish, and as mentioned in our recent blog, applying the ‘Goldilocks’ rule and good scientific citizenship could save a career. There should be no issue with publishing preliminary hypothesis-generating work that is accessible to the research community. The problem is when these findings are publicized to the lay community as having clinical value, particularly for diseases with heavy burdens in poorer populations that raise false hope. Disseminating findings that prove to be meaningless and is reversed over time could have a ‘crying wolf’ effect and undermine public trust in science. 

“More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” (Nature 2016)

It has become increasingly evident that much of the published literature has findings that cannot be reproduced. Reproducibility separates the anecdotal from the real results that should be able to withstand the test of time and replication by the same or other researchers. Nature recently conducted a survey of 1576 researchers and found that more than 50% of researchers could not reproduce their own experiments, and more than 70% could not reproduce other researchers’ experiments. In this survey, more than 60% of respondents cited pressures to publish and selective reporting to be the main contributors to irreproducibility; more than 50% cited low statistical power and fewer still cited variability in reagents and techniques that were difficult to replicate. An overarching problem was also lack of sufficient time to plan and execute protocols, and senior lab members with limited time to train and mentor junior researchers.  Given the importance of mentoring in research, junior researchers who train in such labs may go on to become lab heads, and continue the cycle of productivity without reproducibility.

Misleading or irreproducible research can have far-reaching consequences. Most published research that is preliminary in nature forms the basis of other hypotheses or research questions explored by others researchers with similar interests. If the initial hypothesis-generating research is not sound, and secondary publications expand upon, but do not attempt to validate the original findings, considerable time, money and effort is wasted, careers are affected, and real advances can suffer. In the worst-case scenario, wrong information may form the basis of translational work that enters clinical trials, exposing patients to potentially harmful treatments.  

Industry invests substantially in candidate drug targets sourced from published literature and conference presentations. The validity of candidate drug reports was highlighted by a team of Bayer researchers who retrospectively surveyed all sources of data that contributed to 4 years of in-house validation programs. They found that they were able to reproduce the relevant published findings for only 20–25% of the projects surveyed.

A similar survey by Amgen scientists found that of 53 clinical oncology publications that were deemed ‘landmark’ studies (21 were published by journals with impact factor >20), the findings of only 6 were corroborated by their in-house scientists. The authors of these reports discussed a range of explanations why validation attempts may have failed and the challenges of reproducing published findings, including variations in reagents and experimental models. 

A report by John Arrowsmith at Thomson Reuters on phase II projects from 16 companies, representing ~60% of global R&D spending, showed that success rates had fallen from 28% in 2006–2007, to 18% in 2008–2009. Analysis of 87 phase II failures from 2008–2010, with known reasons for failure, revealed that 51% was due to insufficient efficacy. Although historical trends show that efficacy remains the most common reason for failure, the proportion of efficacy failures decreased by 11% and safety failures decreased by 7% during the period 2013–2015.  

“To do better, insights on reproducibility will be crucial. Laboratory research is of tremendous importance. We should not drown its excellence in a sea of irreproducible results.” (John P.A. Ioannidis 2017)

These reports confirmed the general belief that there was an urgent need for all stakeholders, whether academic institutions, journal reviewers, or funding agencies, to adopt more stringent policies and implement initiatives to improve reproducibility. Basic research has for many decades been the foundation of discoveries that have led to vaccines, new drug developments, and a better understanding of disease processes that contribute to improved health.

Choosing careers in research is indeed a ‘hard slog’ and involves many challenges, and while the thrill of discovery may initially be the driving force, improvements in human health should remain the central focus of health-related research programs.

Despite the current concerns, it is not all ‘doom and gloom’ as data on 2013-2015 phase II & III trials showed that late-phase failure rates are declining and rates of progression at the regulatory review stage had increased. Also recent reports from the Reproducibility Project that aims to independently replicate high-profile research in cancer biology, shows promise.

In our next post we will review some of the methods that can be adopted to improve reproducibility and validation of research findings.  

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.