Evidence Synthesis—Deciding what to believe

Last Updated on September 20th, 2018 by Sharon Johnatty

We often hear the words “evidence-based” thrown around in the media and from politicians on a range of issues like climate or environmental policy, or from those promoting the health benefits of their products. As consumers we put a great deal of stock in ideas and theories that are described as ‘evidence-based’ because it simply has a nice authoritative ring to it. A key question that we should be asking is was the evidence obtained from a comprehensive evidence synthesis approach, i.e. was sufficient effort put into reviewing all available evidence on the topic to draw the stated conclusions?

The dictionary definition of synthesis is “the combination of components or elements to form a connected whole”. Evidence synthesis is the process of pulling together information and knowledge from various sources and disciplines that influences decisions and drives public policy. It is the ‘how, what, why and when’ that goes into ‘evidence-based’ decisions.

The earliest records of evidence-based decisions came from medicine, as documented by James Lind who pioneered The Royal Navy’s approach to dealing with scurvy in the mid-1700s. In fact the British adoption of citrus in their sailors’ diet was one of the factors that gave them superiority over all other naval powers, until this practice became universally adopted. It is interesting to note that Florence Nightingale called statistics ‘the most important science in the world’ as she collected data on sanitation to change hospital practice. She was also an advocate for evidence-based health policies and chastised the British parliament for their scattered approach to health policy.

“You change your laws so fast and without inquiring after results past or present that it is all experiment, seesaw, doctrinaire; a shuttlecock between battledores.” Florence Nightingale’s admonition to the British parliament (1891)

The stated goal of almost every funded research program is doing what is best for the public; therefore a comprehensive overview of outcomes of research needs to be considered to generate policies that are in the best interest of the public. This can be a challenge given the expanding published literature.

As with any process of decision making, the important question is, what constitutes evidence? Is there sufficient information available to systematically analyse and come to a sound conclusion, or are there major gaps in knowledge such that any decision made on the basis of the existing evidence is likely to be unsound and potentially harmful?

Policy makers cannot always predict the outcome of their policies unless similar policies have been successfully implemented elsewhere and under similar conditions. Politicians charged with the business of implementing policies tend to aim for a balance between the best use of public funds and a ‘healthy, wealthy and wise’ agenda (although the ‘wise’ part is often assumed from success with the former two).

Evidence-based policies should therefore rely on the best use of existing evidence. This may require advanced planning and allocation of funds, and many years to implement. However in some instances time may be of the essence— as in responding to a disaster or emergency, in which case both governmental and non-governmental experts from a range of relevant disciplines may be convened to provide advice and manage risks. In healthcare, evidence synthesis influences policy and practice, and given the impact this could have on healthcare costs, the process should ideally be unbiased and accurate and based on all available relevant disciplines.

Various experts have outlined a set of principles that govern evidence synthesis, which if followed, can facilitate development of high-quality evidence. Science of any variety can be contentious and subject to debate depending on the personal and political values of the contender. There will always be topics with a high moral content that lead to disputes, and for which there are no clear-cut right or wrong answers or opinions. Almost all science will at some point transgress on individual sensitivities, and the question remains how to balance this with the greater good of the general population. There is no lack of such examples, whether it is a moral objection to culling pests in order to increase farm productivity, or retaining traditional forms of energy generation versus renewable energy that lower carbon emissions. For this reason, those charged with the task of synthesizing evidence should, where possible, have no personal or financial stake in the outcome, and should stick to unbiased and reputable sources of information.

The following principles have been suggested:

  1. Inclusion of all stakeholders

It would be appropriate to include policy makers if the aim of synthesizing evidence is to advise on a current issue of national importance, for example the economic feasibility of drought-proofing Australian farms. In addition to relevant scientists, community stakeholders who may be the target audience or ‘end-user’ should be involved to add a ‘common sense’ perspective to the issue. This ensures that the question is correctly formulated, and the interpretation of the findings is accurate and not biased in favour of a political agenda. It also brings diversity of opinion and provides several lenses through which the topic is viewed. In contrast, issues like summing up evidence to help drive future policies, such as advanced technologies in artificial intelligence or quantum computing, should be left to the experts.

  1. Rigorous methods

Depending on how urgently the evidence is needed, those involved in evidence synthesis should try to identify all relevant science before deciding on its quality. Public policy based on flawed science can result in costly mistakes, a set-back to progress and a loss of public confidence. Sources of information and reasons for declaring a study as poor evidence should be documented. Where time constraints do not apply, evidence synthesis techniques typically involve systematic reviews, which I have previously written on and can be accessed here. Organizations like Cochrane and The Campbell Collaboration, synthesize evidence to educate the public and inform health-care policy by following predefined methodologies in ways that minimise bias. Such processes are very time-consuming (upwards of 2 years in some cases) but they are well renowned for scientific rigour and generating reports that are comprehensive and reputable.

  1. Transparency

Sources of evidence, databases used, search terms, and how evidence is graded should all be publicly available and transparent to end-users. Although study methodology should follow a pre-defined process, areas of difficulty may arise in whether to include or exclude certain studies. Accounts should be kept of decisions, the reason for the difficulty or disagreement, and why a consensus could not be reached, as this may be important in future updates or policy debates.

  1. Open Access

Evidence summaries in plain language that is accessible and available online is critical to their wider acceptance by relevant communities, policy makers and the general population. Depending on the range of potential end-users, multiple reports of the synthesized evidence may be necessary. Infographics or interactive online demonstrations that inform and educate the public will go a long way in gaining support for, and successful implementation of policies informed by this process. Timing is also critical, and updating the knowledge base regularly on topics of local and regional importance will reduce reliance on inaccurate or outdated information in an emergency.

On a global scale, evidence synthesis is critical to coordinated responses to disease outbreaks. Costs of such efforts are often borne by countries with the ability to fund them, and stakeholders need to be convinced to participate in a unified effort. During the 2014 Ebola epidemic in West Africa, SAGE convened a wide range of experts from around the world. It needs to be irrefutable that the universal benefit of such an undertaking far outweighs the self-interest of any particular country.

A limitation of the current best-practice of systematic reviews is the likelihood that existing studies are of low quality or highly variable in their findings to produce reliable results. An alternative is subject-wide evidence synthesis, which involves extracting and collating relevant information from many different sources. This has been done on a project called Conservation Evidence, which provides summary information on the effects of conservation interventions for all species and habitats worldwide. Although this approach is quite different from systematic reviews, it provides a valuable searchable database that can be used in combination with systematic review methods to address new research questions on conservation.

The subject-wide approach is not limited to conservation or environmental sciences, but may be useful in public health questions where a particular outcome is so rare that even large well-powered studies are not sufficient to shed light on risk factors. I recently did an analysis of uterine cancer among women who were treated with tamoxifen for prior breast cancer. Although the risks associated with tamoxifen treatment are well documented, our data suggested that a high proportion of women treated with tamoxifen subsequently developed a rather nasty type of uterine tumour known to have very poor prognosis. This has also been documented in several case studies. While case studies tend to be viewed as anecdotal and insufficient evidence to guide health policy changes, it does suggest the need for improved surveillance of women treated with tamoxifen.

Decisions and critical assertions made by public officials and politicians that impact on matters that affect people’s lives—whether it is indigenous affairs, climate change, energy policies, or healthcare—cannot be done on a whim or in a rush to placate political constituencies and win votes. The importance of a comprehensive assessment of all evidence to ensure that polices are indeed evidence-based, should not be side-lined in favour of ‘quick and dirty’ alternatives.

At SugarApple Communications we can help you find the best way to analyse and interpret your important data, and communicate it to your intended audience. Get in touch today and let’s talk.

← Back to All Articles

Unfog the science…ensure quality, clarity and accuracy.