Last Updated on October 9th, 2017 by

Healthcare research is becoming increasingly more accessible as the level of reporting and commentary rises, both via established news sources and on health-related internet sites and private blogs. The public appetite for health information is growing as evidenced by the popularity and uptake of such information, regardless of whether or not it comes from a trustworthy source. A well-written narrative review, or a carefully designed systematic review and meta-analysis, is important to summarizing healthcare research. 

We are bombarded almost daily with new and often changing reports about the latest research — “drink red wine”…“don’t drink red wine”…“drink red wine once or twice per week”… and so on. As we discussed in an earlier blog, overly ambitious claims may be contributing to a general mistrust, within society, of medical advice and guidance, even when the supporting data is reliable.

A recent commentary in Nature suggested that more collaboration was needed between computational experts and population-health researchers in order to synthesize health evidence as more and more data become available. Reviews that aim to summarize the available research to provide the ‘bottom line’ are a valuable resource for clinicians, researchers and the community as a whole. Such reviews can be conducted in a number of ways – from narrative reviews or opinionated summaries to more analytical, protocol-defined systematic reviews capable of providing summary estimates that can guide health policy.

Narrative reviews are traditional literature reviews that describe and summarize current knowledge on a specific topic or area of research. Authors of published narrative reviews should ideally have spent their careers working in that field. They generally use informal and subjective methods to screen and select studies to include, and apply their expertise on the topic to interpret and summarize these studies to provide a snapshot of the current research. Narrative reviews are largely qualitative, and tend to rationalize the diversity of information around the topic into a single coherent article, which critically evaluates the research, highlights gaps in the existing knowledge, and may even recommend areas that need more work. Authors should ideally also include a summary of the search criteria utilized, the inclusion or exclusion criteria, and their rationale for selecting them.

Narrative reviews may be voluntary, or commissioned by journals from leading researchers, on topics of current interest, with the purpose of obtaining consensus statements and perspectives on where the current research lies, and also to evaluate trends in research to direct future publications.

Systematic reviews, in contrast, aim to identify, evaluate and summarize all relevant studies on a clearly formulated question, using systematic methods according to a strict, pre-defined protocol. A systematic review should combine the results of selected individual studies to obtain a single more reliable overall estimate. The methodology involves a rigorous approach to ensure all possible, relevant research is considered, thereby minimizing the risk of bias. The criteria for inclusion should be transparent, and reasons for inclusion and exclusion clearly stated. All methods including search terms, criteria for selection, appraisal of the studies, and statistical methods, should be reported to allow study replication if needed.

The trustworthiness of a systematic review depends on a priori planning (both with regard to study selection and analytic approaches), careful documentation of a planned protocol agreed upon by those involved, and strict adherence to the set protocol. This avoids subjective or arbitrary decisions with respect to data extraction, or bias and selective reporting of findings, which could negatively impact healthcare decision-making.

Authors of systematic reviews should include commentary on the quality of each input study, an interpretation of their summary result and how best it can be used in clinical practice, their confidence in the result, and how they differ from those of other published studies. The old adage ‘rubbish in – rubbish out’ holds true for systematic reviews, and careful study selection is pivotal to their utility.

“Meta-analysis is the analysis of analyses … the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” (Glass 1976)

Meta-analysis is the statistical technique used to combine and summarize the results of quantitative studies to provide a single more precise estimate. It is a common misconception that the terms ‘systematic review’ and ‘meta-analysis’ are synonymous. They aren’t. Some systematic reviews involve studies with results that cannot be pooled statistically. It would, therefore, be inappropriate to apply meta-analysis to such studies.

The use of meta-analysis is not limited only to systematic reviews. Meta-analysis can be applied to small studies that address similar questions, and is particularly valuable where the individual estimates are inconclusive due to the small sample size. Combining these studies in a meta-analysis can improve estimates by increasing the sample size and therefore the statistical power to detect a true effect.

Critics of meta-analysis argue that this technique amounts to combining ‘apples and oranges’ that will generate an essentially meaningless estimate. These are valid concerns for small observational studies that have considerable study heterogeneity, or where there is variation between the results of individual studies. Ideally, meta-analysis should only be applied to studies with sufficiently similar design, study population, intervention or comparison, and outcome measures. The extent of variation or heterogeneity between studies can be assessed using a number of available techniques.  Authors of a systematic review should report the p-value for formal statistical tests for study heterogeneity, and provide some discussion on this in the context of the overall utility of their findings.

Much work has been done to standardize protocols for the conduct and reporting of systematic reviews and meta-analyses. The most widely endorsed and accepted is PRISMA, which was published in 2009. These and other guidelines on reporting health research studies can be found on the EQUATOR Network, an umbrella organization of multiple stakeholders with the goal of improving the quality of research and research publications. The Cochrane Library and Campbell Collaboration regularly publish systematic reviews and their protocols. Affiliated with Cochrane and EQUATOR is PROSPERO, an international database that prospectively registers systematic review protocols, which was launched through the University of York (UK) in 2011. By 2016, the number of prospective registrations reached 20,000 records from 107 countries.

Narrative reviews, systematic reviews and meta-analyses are all useful ways to evaluate and summarize large numbers of studies on the same subject. However, they are only as good as the studies they summarize, and the relative merits and limitations of each should be appreciated. This is critical when reviews are used to guide medical decisions and develop clinical guidance. Systematic reviews that follow clearly defined protocols and adhere to the guidelines for good conduct and reporting practices are a gold standard that provides the most reliable estimates.

In an upcoming article, we will discuss ways to identify characteristics of a sufficiently credible, well-written systematic review that ticks the boxes for good reporting practice.

At SugarApple Communications we can help you find the best way to communicate with your intended audience and assist with writing, editing and statistics. Get in touch today and let’s talk.