Miranda Cumpston, Toby Lasserson, Ella Flemyng, Matthew J Page
Key Points:
- Clear reporting of a systematic review allows readers to evaluate the rigour of the methods applied, and to interpret the findings appropriately. Transparency can facilitate attempts to verify or reproduce the results, and make the review more usable for health care decision makers.
- The target audience for Cochrane Reviews is people making decisions about health care, including healthcare professionals, consumers and policy makers. Cochrane Reviews should be written so that they are easy to read and understand by someone with a basic sense of the topic who may not necessarily be an expert in the area.
- Cochrane Protocols and Reviews should comply with the PRISMA 2020 and PRISMA for Protocols reporting guidelines.
- Guidance on the composition of plain language summaries of Cochrane Reviews is also available to help review authors specify the key messages in terms that are accessible to consumers and non-expert readers.
- Review authors should ensure that reporting of objectives, important outcomes, results, caveats and conclusions is consistent across the main text, the abstract, and any other summary versions of the review (e.g. plain language summary).
Cite this chapter as: Cumpston M, Lasserson T, Flemyng E, Page MJ. Chapter III: Reporting the review [last updated August 2023]. In: Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane, 2024. Available from cochrane.org/handbook.
III.1 Introduction
The effort of undertaking a systematic review is wasted if review authors do not report clearly what they did and what they found (Glasziou et al 2014). Clear reporting enables others to replicate the methods used in the review, which can facilitate attempts to verify or reproduce the results (Page et al 2018). Transparency can also make the review more usable for healthcare decision makers. For example, clearly describing the interventions assigned in the included studies can help users determine how best to deliver effective interventions in practice (Hoffmann et al 2017). Also, comprehensively describing the eligibility criteria applied, sources consulted, analyses conducted, and post-hoc decisions made, can reduce uncertainties in assessments of risk of bias in the review findings (Whiting et al 2016). For these reasons, transparent reporting is an essential component of all systematic reviews.
Surveys of the transparency of published systematic reviews suggest that many elements of systematic reviews could be reported better. For example, Nguyen and colleagues evaluated a random sample of 300 systematic reviews of interventions indexed in bibliographic databases in November 2020 (Nguyen et al 2022). They found that in at least 20% of the reviews there was no information about the years of coverage of the search, the methods used to collect data and appraise studies, or the funding source of the review. Less than half of the reviews provided information on a protocol or registration record for the review. However, Cochrane Reviews, which accounted for 3% of the sample, had more complete reporting than all other types of systematic reviews.
Possible reasons why more complete reporting of Cochrane Reviews has been observed include the use of software (RevMan, /online-learning/core-software-cochrane-reviews/revman) and strategies in the editorial process that promote good reporting. RevMan includes many standard headings and subheadings which are designed to prompt Cochrane Review authors to document their methods and results clearly.
Cochrane Reviews of interventions should adhere to the PRISMA 2020 (Preferred Reporting Items for Systematic reviews and Meta-Analysis) reporting guideline; see http://www.prisma-statement.org/. PRISMA is an evidence-based, minimum set of items for reporting systematic reviews and meta-analyses to ensure the highest possible standard for reporting is met. Extensions to PRISMA and additional reporting guidelines for specific areas of methods are cited in the relevant sections below.
Cochrane’s Methodological Expectations of Cochrane Intervention Reviews (MECIR) detail standards for the conduct of Cochrane Reviews of interventions. They provide expectations for the general methodological approach to be followed from designing the review up to interpreting the findings at the end. There is a good reason to distinguish between conduct (MECIR) and reporting (PRISMA): good conduct does not necessarily lead to good reporting, good reporting cannot improve poor conduct, and poor reporting can obscure good or poor conduct of a review. The MECIR expectations of conduct are embedded in the relevant chapters of this Handbook and authors should adhere to MECIR throughout the development of their systematic review. MECIR conduct guidance for updates of Cochrane Reviews of interventions are presented in Chapter IV. For the latest version of all MECIR conduct guidance, readers should consult the MECIR web pages, available at https://methods.cochrane.org/mecir.
This chapter is built on reporting guidance from PRISMA 2020 (Page et al 2021a, Page et al 2021b) and is divided into sections for Cochrane Review protocols (Section III.2) and new Cochrane Reviews (Section III.3). Many of the standard headings recommended for use in Cochrane Reviews are referred to in this chapter, although the precise headings available in RevMan may be amended as new versions are released. New headings can be added and some standard headings can be deactivated; if the latter is done, review authors should ensure that all information expected (as outlined in PRISMA 2020) is still reported somewhere in the review.
III.2 Reporting of protocols of new Cochrane Reviews
Preparing a well-written review protocol is important for many reasons (see Chapter 1). The protocol is a public record of the question of interest and the intended methods before results of the studies are fully known. This helps readers to judge how the eligibility criteria of the review, stated outcomes and planned methods will address the intended question of interest. It also helps anyone who evaluates the completed review to judge how far it fulfilled its original objectives (Lasserson et al 2016). Investing effort in the development of the review question and planning of methods also stimulates review authors to anticipate methodological challenges that may arise, and helps minimize potential for non-reporting biases by encouraging review authors to publish their review and report results for all pre-specified outcomes (Shamseer et al 2015).
See the Introduction and Methods sections of PRISMA 2020 for the reporting items relevant to protocols for new Cochrane Reviews. All these items are also covered in PRISMA for Protocols, an extension to the PRISMA guidelines for the reporting of systematic review protocols (Moher et al 2015, Shamseer et al 2015). They include guidance for reporting of the:
- Background;
- Objectives;
- Criteria for considering studies for inclusion in the review;
- Search methods for identification of studies (e.g. a list of all sources that will be searched, a complete search strategy to be implemented for at least one database);
- Data collection and analysis (e.g. types of information that will be sought from reports of included studies and methods for obtaining such information, how risk of bias in included studies will be assessed, and any intended statistical methods for combining results across studies); and
- Other information (e.g. acknowledgements, contributions of authors, declarations of interest, and sources of support).
These sections correspond to the same sections in a completed review, and further details are outlined in Section III.3.
The required reporting items have been incorporated into a template for protocols for Cochrane Reviews, which is available in Cochrane’s review production tool, RevMan (see the RevMan Knowledge Base). If using the template, authors should carefully consider the methods that are appropriate for their specific review and adapt the template where required.
One key difference between a review protocol and a completed review is that the Methods section in a protocol should be written in the future tense. Because Cochrane Reviews are updated as new evidence accumulates, methods outlined in the protocol should generally be written as if a suitably large number of studies will be identified to allow the objectives to be met (even if this is assumed to be unlikely the case at the time of writing).
PRISMA 2020 reflects the minimum expectations for good reporting of a review protocol. Further guidance on the level of planning required for each aspect of the review methods and the detailed information recommended for inclusion in the protocol is given in the relevant chapters of this Handbook.
III.3 Reporting of new Cochrane Reviews
The main text of a Cochrane Review should be succinct and readable. Although there is no formal word limit for Cochrane Reviews, review authors should consider 10,000 words a maximum for the main text of the review unless there is a special reason to write a longer review, such as when the question is unusually broad or complex.
People making decisions about health care are the target audience for Cochrane Reviews. This includes healthcare professionals, consumers and policy makers, and reviews should be accessible to these audiences. Cochrane Reviews should be written so that they are easy to read and understand by someone with a basic sense of the topic who is not necessarily an expert in the area. Some explanation of terms and concepts is likely to be helpful, and perhaps even essential. However, too much explanation can detract from the readability of a review. Simplicity and clarity are also vital to readability. The readability of Cochrane Reviews should compare to that of a well-written article in a general medical journal.
Review authors should ensure that reporting of objectives, outcomes, results, caveats and conclusions is consistent across the main text, the tables and figures, the abstract, and any other summary versions of the review (e.g. ‘Summary of findings’ table and plain language summary). Although this sounds simple, it can be challenging in practice; authors should review their text carefully to ensure that readers of a summary version are likely to come away with the same overall understanding of the conclusions of the review as readers accessing the full text.
Plagiarism is not acceptable and all sources of information should be cited (for more information see the Cochrane Library editorial policy on plagiarism). Also, the unattributed reproduction of text from other sources should be avoided. Quotes from other published or unpublished sources should be indicated and attributed clearly, and permission may be required to reproduce any published figures.
PRISMA 2020 provides the main reporting items for new Cochrane Reviews. A template for Cochrane Reviews of interventions is available that incorporates the relevant reporting guidance from PRISMA 2020. The template is available in RevMan to facilitate author adherence to the reporting guidance via the RevMan Knowledge Base. If using the template, authors should consider carefully the methods that are appropriate for their specific review and adapt the template where required. In the remainder of this section we summarize the reporting guidance relating to different sections of a Cochrane Review.
III.3.1 Abstract
All reviews should include an abstract of not more than 1000 words, although in the interests of brevity, authors should aim to include no more than 700 words, without sacrificing important content. Abstracts should be targeted primarily at healthcare decision makers (clinicians, consumers and policy makers) rather than just to researchers.
Terminology should be reasonably easy to understand for a general rather than a specialist healthcare audience. Abbreviations should be avoided, except where they are widely understood (e.g. HIV). Where essential, other abbreviations should be spelt out (with the abbreviations in brackets) on first use. Names of drugs and interventions that can be understood internationally should be used wherever possible. Trade or brand names should not be used and generic names are preferred.
Abstracts of Cochrane Reviews are made freely available on the internet and published in bibliographic databases that index the Cochrane Database of Systematic Reviews (e.g. MEDLINE, Embase). Some readers may be unable to access the full review, or the full text may not have been translated into their language, so abstracts may be the only source they have to understand the review results (Beller et al 2013). It is important therefore that they can be read as stand-alone documents. The abstract should summarize the key methods, results and conclusions of the review. An abstract should not contain any information that is not in the main body of the review, and the overall messages should be consistent with the conclusions of the review.
Abstracts for Cochrane Reviews of interventions should follow the PRISMA 2020 for Abstracts checklist (Page et al 2021b). Each abstract should include:
- Rationale (a concise summary of the rationale for and context of the review);
- Objectives (of the review);
- Search methods (including an indication of databases searched, and the date of the last search for which studies were fully incorporated);
- Eligibility criteria (including a summary of eligibility criteria for study designs, participants, interventions and comparators);
- Outcomes;
- Risk of bias (methods used to assess risk of bias);
- Synthesis methods (methods used to synthesize results, especially any variations on standard approaches);
- Included studies (total number of studies and participants and a brief summary of key characteristics);
- Results of syntheses (including the number of studies and participants for each outcome, a clear statement of the direction and magnitude of the effect, the effect estimate and 95% confidence interval if meta-analysis was used, and the GRADE assessment of the certainty of the evidence. The results should contain the same outcomes as found in other summary formats such as the plain language summary and ‘Summary of findings’ table, including those for which no studies reported the outcome and those that are not statistically significant. This section should also provide a brief summary of the limitations of the evidence included in the review);
- Authors’ conclusions (including implications both for practice and for research);
- Funding (primary source of funding for the review); and
- Registration (registration name and number and/or DOIs of previously published protocols and versions of the review, if applicable).
III.3.2 Plain language summary
A Cochrane Plain language summary is a stand-alone summary of the systematic review. Like the Abstract, the Plain language summary may be read alone, and its overall messages should be consistent with the conclusions in the full review.
The Plain language summary should convey clearly the questions and key findings of the review, using language that can be understood by a wide range of non-expert readers. The summary should use words and sentence structures that are easy to understand, and should avoid technical terms and jargon where possible. Any technical terms used should be explained. The audience for Plain language summaries may include people with a health condition, carers, healthcare workers or policy makers. Readers may not have English as their first language. Cochrane Plain language summaries are frequently translated, and using plain language is also helpful for translators.
Writing in plain language is a skill that is different from writing for a scientific audience. Full guidance and a template are available as online supplementary material to this chapter. Authors are strongly encouraged to use this guidance to ensure good practice and consistency with other summaries in the Cochrane Library. It may also be helpful to seek assistance for this task, such as asking someone with experience in writing in plain language for a general audience for help, or seeking feedback on the draft summary from a consumer or someone with little knowledge of the topic area.
III.3.3 Background and Objectives
Well-formulated review questions occur in the context of an already-formed body of knowledge. The Background section should address this context, including a description of the condition or problem of interest. It should help clarify the rationale for the review, and explain why the questions being addressed are important. It should be concise (generally under 1000 words) and be understandable to the users of the intervention(s) under investigation.
It is important that the eligibility criteria and other aspects of the methods, such as the comparisons used in the synthesis, build on ideas that have been developed in the Background section. For example, if there are uncertainties to be explored in how variation in setting, different populations or type of intervention influence the intervention effect, then it would be important to acknowledge these as objectives of the review, and ensure the concepts and rationale are explained.
The following three standard subheadings in the Background section of a Cochrane Review are intended to facilitate a structured approach to the context and overall rationale for the review.
- Description of the condition: A brief description of the condition being addressed, who is affected, and its significance, is a useful way to begin the review. It may include information about the biology, diagnosis, prognosis, prevalence, incidence and burden of the condition, and may consider equity or variation in how different populations are affected.
- Description of the intervention and how it might work: A description of the experimental intervention(s) should place it in the context of any standard or alternative interventions, remembering that standard practice may vary widely according to context. The role of the comparator intervention(s) in standard practice should also be made clear. For drugs, basic information on clinical pharmacology should be presented where available, such as dose range, metabolism, selective effects, half-life, duration and any known interactions with other drugs. For more complex interventions, such as behavioural or service-level interventions, a description of the main components should be provided (see Chapter 17). This section should also provide theoretical reasoning as to why the intervention(s) under review may have an impact on potential recipients, for example, by relating a drug intervention to the biology of the condition. Authors may refer to a body of empirical evidence such as similar interventions having an impact on the target recipients or identical interventions having an impact on other populations. Authors may also refer to a body of literature that justifies the possibility of effectiveness. Authors may find it helpful to use a logic model (Kneale et al 2015) or conceptual framework to illustrate the proposed mechanism of action of the intervention and its components. This will also provide review authors with a framework for the methods and analyses undertaken throughout the review to ensure that the review question is clearly and appropriately addressed. More guidance on considering the conceptual framework for a particular review question is presented in Chapter 2 and Chapter 17.
- Why it is important to do this review: Review authors should explain clearly why the questions being asked are important. Rather than justifying the review on the grounds that there are known eligible studies, it is more helpful to emphasize what aspects of, or uncertainties in, the accumulating evidence base now justify a systematic review. For example, it might be the case that studies have reached conflicting conclusions, that there is debate about the evidence to date, or that there are competing approaches to implementing the intervention.
Immediately following the Background section of the review, review authors should declare the review objectives. They should begin with a precise statement of the primary objective of the review, ideally in a single sentence. Where possible the style should be of the form “To assess the effects of [intervention or comparison] for [health problem] for/in [types of people, disease or problem and setting if specified]”. This might be followed by a series of secondary objectives relating to different participant groups, different comparisons of interventions or different outcome measures. If relevant, any objectives relating to the evaluation of economic or qualitative evidence should be stated. It is not necessary to state specific hypotheses.
III.3.4 Methods
The Methods section in a completed review should be written in the past tense, and should describe what was done to obtain the results and conclusions of the current review.
Review authors are expected to cite their protocol to make it clear that there was one. Often a review is unable to implement all the methods outlined in the protocol. For example, planned investigations of heterogeneity (e.g. subgroup analyses) and small-study effects may not have been conducted because of an insufficient number of studies. Authors should describe and explain all amendments to the prespecified methods in the main Methods section.
The Methods section of a Cochrane Review includes five main subsections, within which are a series of standard headings to guide authors in reporting all the relevant information. See Sections III.3.4.1, III.3.4.2, III.3.4.3, III.3.4.4, and III.3.4.5 for a summary of content recommended for inclusion under each subheading.
III.3.4.1 Criteria for considering studies for this review
Review authors should declare all criteria used to decide which studies are included in the review. Doing so will help readers understand the scope of the review and recognize why particular studies they are aware of were not included. Eligible study designs should be described, with a focus on specific features of a study’s design rather than design labels (e.g. how groups were formed, whether the intervention was assigned to individuals or clusters of individuals) (Reeves et al 2017). Review authors should describe eligibility criteria for participants, including any restrictions based on age, diagnostic criteria, location and setting. If relevant, it is useful to describe how studies including a subset of relevant participants were addressed (e.g. when children up to the age of 16 years only were eligible but a study included children up to the age of 18 years). Eligibility criteria for interventions and comparators should be stated also, including any criteria around delivery, dose, duration, intensity, co-interventions and characteristics of complex interventions. The rationale for all criteria should be clear, including the eligible study designs.
Typically, studies should not be excluded from a review solely because no outcomes of interest were reported, because failure to report an outcome does not mean it was not assessed (Dwan et al 2017). However, on occasion it will be appropriate to include only studies that measured particular outcomes. For example, a review of a multi-component public health intervention promoting healthy lifestyle choices, focusing on reduction in smoking prevalence, might legitimately exclude studies that do not measure any smoking outcomes. Review authors should specify if measurement of a particular outcome was used as an eligibility criterion for the review, and justify why this was done.
Further guidance on planning eligibility criteria is presented in Chapter 3.
III.3.4.2 Outcome measures
Review authors should specify the critical and important outcomes of interest to the review, and define acceptable ways of measuring them. The review’s important outcomes should normally reflect at least one potential benefit and at least one potential harm.
For each listed outcome or outcome domain, it should be clear which specific outcomes, measures or tools will be considered together and combined for the purposes of synthesis. For example, for the outcome of depression, a series of measurement tools for depression symptoms may be listed. It should be explicitly stated whether they will be synthesized together as a single outcome (depression), or presented as a series of separate syntheses for each tool. Any categories of time that will be used to group outcomes for synthesis should also be defined, e.g. short term (up to 1 month), medium term (> 1 month to 12 months), and long term (> 12 months). Additional guidance on grouping of outcomes for synthesis is included in Chapter 3, and in the InSynQ (Intervention Synthesis Questions) reporting guideline (https://InSynQ.info).
III.3.4.3 Search methods for identification of studies
It is essential that users of systematic reviews are given an opportunity to evaluate the methods used to identify studies for inclusion. Such an evaluation is possible when review authors report their search methods comprehensively. This involves specifying all sources consulted, including databases, trials registers, websites, and a list of individuals or organizations contacted. If particular journals were handsearched, this should be noted. Any specific methods used to develop the search strategy, such as automated text analysis or peer review, should also be noted, including methods used to translate the search strategy for use in different databases. Specifying the dates of coverage of all databases searched and the date of the last search for which studies were fully incorporated can help users determine how up to date the review is. Review authors should also declare any limits placed on the search (e.g. by language, publication date or publication format).
To facilitate replication of a search, review authors should include in the supplementary material the exact search strategy (or strategies) used for each database, including any limits and filters used. Search strategies can be exported from bibliographic databases, and these should be copied and pasted instead of re-typing each line, which can introduce errors.
See Chapter 4 for guidance on search methods. An extension to the PRISMA statement for reporting of literature searches is also available (Rethlefsen et al 2021).
III.3.4.4 Data collection and analysis
Cochrane Reviews include several standard subheadings to enable a structured, detailed description of the methods used for data collection and analysis. Additional headings should be included where appropriate to describe additional methods implemented in the review, e.g. those specific to the analysis of qualitative or economic evidence.
Selection of studies: There should be a description of how the eligibility criteria were applied, from screening of search results through to the final selection of studies for inclusion in the review. The number of people involved at each stage of the process should be stated, such as two authors working independently, along with an indication of how any disagreements were resolved. Any automated processes, software tools or crowdsourcing used to support selection should be noted. See Chapter 4 for guidance on the study selection process.
Data collection and management: Review authors should specify how data were collected for the included studies. This includes describing the number of people involved in data collection, whether they worked independently, how any disagreements were resolved, and whether standardized data collection forms were used (and if so, whether they were piloted in advance). Any software tools used in data collection should be cited, as well as any checklists such as TIDieR for the description of interventions (Hoffmann et al 2017), TIDieR PHP for population health and policy interventions (Campbell et al 2018), or TACIT for identifying conflicts of interest (https://tacit.one/). If study authors or sponsors were contacted to obtain missing information or to clarify the information available, this should be stated.
RevMan allows authors to directly import some types of data (including study results and risk of bias assessments). To facilitate the import, it is recommended that Cochrane authors consider the required format of data import files to inform their data extraction forms. See documentation in the RevMan Knowledge Base.
A brief description of the data items (e.g. participant characteristics, intervention details) extracted from each report is recommended. If methods for transforming or processing data in preparation for analysis were necessary (e.g. converting standard errors to standard deviations, extracting numeric data from graphs), these methods should be described.
Additional information about the outcomes to be collected is helpful to include, including a description of how authors handled multiplicity, such as where a single study reports more than one similar outcome measure or measurement time point eligible for inclusion in the same review, requiring a method or decision rule to select between eligible results.
See Chapter 3 for guidance on selecting outcomes, and Chapter 5 for guidance on data collection.
Risk of bias assessment in included studies: There should be a description of the approach used to assess risk of bias in the included studies. This involves specifying the risk-of-bias tool(s) used, how many authors were involved in the assessment, how disagreements were resolved, and how the assessments were incorporated into the analysis or interpretation of the results. The preferred bias assessment tools for Cochrane review authors are RoB 2 for RCTs and ROBINS -I for non-randomized studies (described in Chapter 8 and Chapter 25).
When using either of these tools, some specific information is needed in this section of the Methods. Authors should specify the outcome measures and timepoints assessed (often the same prespecified outcomes were considered in the GRADE assessment and included in summary versions of the review, see Chapter 3, Section 3.2.4.2); and the effect of interest the author team assessed (either the effect of assignment to the intervention, or the effect of adhering to the intervention). Authors should also specify how overall judgements were reached, both across domains for an individual result and across multiple studies included in a synthesis.
Cochrane has developed checklists for reporting risk of bias methods in protocols and completed reviews for authors using the RoB 2 tool (https://methods.cochrane.org/risk-bias-2) and the ROBINS-I tool (https://methods.cochrane.org/robins-i). See Chapter 7 for further guidance on study risk-of-bias assessment. Authors who have used the original version of the RoB tool (from 2008 or 2011) should refer to guidance for reporting the risk of bias in version 5.2 of the Cochrane Handbook for Systematic Reviews of Interventions (available here).
Measures of the treatment effect: The effect measures used by the review authors to describe results in any included studies or meta-analyses (or both) should be stated. Examples of effect measures include the odds ratio (OR), risk ratio (RR) and risk difference (RD) for dichotomous data; the mean difference (MD) and standardized mean difference (SMD) for continuous data; and hazard ratio for time-to-event data. Note that some non-randomized study designs require different effect estimates, and these should be specified if such designs are included in the review (e.g. interrupted time series commonly measure the change in level and change in slope). See Chapter 6 for more guidance on effect measures.
Unit of analysis issues: If the review includes study designs that can give rise to a unit-of-analysis error (when the number of observations in an analysis does not match the number of units randomized), the approaches taken to address these issues should be described. Studies that can give rise to unit-of-analysis errors include crossover trials, cluster-randomized trials, studies where interventions are assigned to multiple parts of the body of the same participant, and studies with multiple intervention groups where more than two groups are included in the same meta-analysis. See Chapter 23 for guidance on handling unit-of-analysis issues.
Dealing with missing data: Review authors may encounter various types of missing data in their review. For example, there may be missing information that has not been reported by the included studies, such as information about the methods of the included studies (e.g. when the method of randomization is not reported, which may be addressed in the risk of bias assessment); missing statistics (e.g. when standard deviations of mean scores are not reported, where missing statistics may be calculated from the available information or imputed); or non-reporting of outcomes (which may represent a risk of bias due to missing results). Missing data may also refer to cases where participants in the included primary studies have withdrawn or been lost to follow-up, or have missing measurements for some outcomes, which may be considered and addressed through risk of bias assessment. Any strategies used to deal with missing data should be reported, including any attempts to obtain the missing data. See Chapter 10 for guidance on dealing with missing data.
Reporting bias assessment: Any methods used to assess the risk of bias due to missing results should be described. Such methods may include consideration of the number of studies missing from a synthesis due to selective non-reporting of results, or investigations to assess small-study effects (e.g. funnel plots), which can arise from the suppression of small studies with ‘negative’ results (also called publication bias). If relevant, any tools or checklists used (such as ROB-ME, https://www.riskofbias.info/welcome/rob-me-tool) should be cited.See Chapter 13 for a description of methods for assessing risk of bias due to missing results in a synthesis.
Synthesis methods: Reviews may address multiple research questions (‘synthesis questions’). For example, a review may be interested in the effects of an intervention in children or adults, or may wish to investigate the effects of different types of exercise interventions. Each comparison to be made in the synthesis should be specified in enough detail to allow a reader to replicate decisions about which studies belong in each synthesis, and the rationale for the comparisons should be clear. Comparisons for synthesis can be defined using the same PICO characteristics that are used to define the eligibility criteria for including studies in the review. See Chapter 3 for guidance on defining the ‘PICO for each synthesis’. Further guidance is available in the InSynQ (Intervention Synthesis Questions) tool for planning and reporting synthesis questions (https://InSynQ.info).
Review authors should then describe the methods used for synthesizing results across studies in each comparison (e.g. meta-analysis, network meta-analysis or other methods). Where data have been combined in statistical software external to RevMan, authors should reference the software, commands and settings used to run the analysis. See Chapter 10 for guidance on undertaking meta-analysis, Chapter 11 for guidance on undertaking network meta-analysis, and Chapter 12 for a description of other synthesis methods. An extension to the PRISMA statement for reporting network meta-analyses is available for reviews using these methods (Hutton et al 2015).
Where meta-analysis is planned, details should be specified of the meta-analysis model (e.g. fixed-effect or random-effects), the specific method used (e.g. Mantel Haenszel, inverse variance, Peto), and a rationale presented for the options selected. Review authors should also describe their approach to identifying or quantifying statistical heterogeneity (e.g. visual inspection of results, a formal statistical test for heterogeneity, I2, Tau2, or prediction interval). See Chapter 10 for guidance on assessment of heterogeneity.
Where meta-analysis is not possible, any other synthesis methods used should be described explicitly, including the rationale for the methods selected. It is common for these methods to be insufficiently described in published reviews (Campbell et al 2019, Cumpston et al 2023), and general terms such as ‘narrative synthesis’ do not provide appropriate detail about the specific methods used. In addition to detailed guidance in Chapter 12, a reporting guideline for Synthesis Without Meta-analysis (SWiM) has been developed and should be considered in addition to MECIR for reporting these methods(Campbell et al 2020).
For whichever synthesis methods are used, the structure of tables and plots used to visually display results should also be specified, including a rationale for the options selected (see Section III.3.5.4).
Investigations of heterogeneity and subgroup analysis: If subgroup analyses or meta-regression were performed, review authors should specify the potential effect modifiers explored, the rationale for each, whether they were identified before or after the results were known, whether they were based on between-study or within-study subgroups, and how they were compared (e.g. using a statistical test for interaction). See Chapter 10 for more information on investigating heterogeneity. If applicable, review authors should specify which equity-related characteristics were explored.
Sensitivity analysis: If any sensitivity analyses were performed to explore the robustness of meta-analysis results, review authors should specify the basis of each analysis (e.g. removal of studies at high risk of bias, imputing alternative estimates of missing standard deviations). See Chapter 10 for more information on sensitivity analyses.
Certainty of the evidence assessment: Review authors should describe methods for summarizing the findings of the review, and assessing the certainty of the body of evidence (e.g. using the GRADE approach). The domains to be assessed should be stated, including any thresholds used to downgrade the certainty of the evidence, such as risk of bias assessment, levels of unexplained heterogeneity, or key factors for assessing directness. Who conducted the GRADE assessment should be stated, including whether two authors assessed GRADE independently and how disagreements were resolved.
Review authors should also indicate which populations, interventions, comparisons and outcomes are addressed in ‘Summary of findings’ tables, specifying up to seven prioritized critical or important outcomes to be included. Authors should note what they considered to be a minimally important difference for each outcome.
Any specific language used to describe results in the context of the GRADE assessment should be explained, such as using the word “probably” for to moderate-certainty evidence, and “may” in relation to low-certainty evidence (see Chapter 15, Section 15.6.4).
For more details on completing ‘Summary of findings’ tables and using the GRADE approach, see Chapter 14.
III.3.4.5 Consumer Involvement
Cochrane follows the ACTIVE (Authors and Consumers Together Impacting on eVidencE) framework to help review authors have meaningful involvement in their systematic reviews (Pollock et al 2017). Review authors should report on their methods for involving consumers in their review, including the authors’ general approach to involvement; the level of involvement and the roles of the consumers involved; the stage in the review process when involvement occurs; and any formal research methods or techniques used.
Other stakeholders may also be involved in the systematic reviews, such as health care providers, policy makers and other decision makers. Where other stakeholders are involved, this should also be described.
If review authors did not involve consumers or other stakeholders, this should be stated.
III.3.5 Results
A narrative summary of the results of a Cochrane Review should be provided under the three standard subheadings in the Results section (see Sections III.3.5.1, III.3.5.2 and III.3.5.3 for a summary of content recommended for inclusion under each subheading). Details about the effects of interventions (including summary statistics and effect estimates for each included study and for synthesis) can be presented in various tables and figures (see Section III.3.5.4).
III.3.5.1 Description of studies
The results section should start with a summary of the results of the search (for example, how many references were retrieved by the electronic searches, how many were evaluated after duplicates were removed, how many were considered as potentially eligible after screening, and how many were included). Review authors are expected to include a PRISMA-type flow diagram demonstrating the flow of studies throughout the selection process (Page et al 2021b). Such flow diagrams can be created within RevMan.
To help readers determine the completeness and applicability of the review findings in relation to the review question, as well as how studies are grouped for synthesis within the review, authors should describe the characteristics of the included studies. In the Results section, a brief narrative summary of the included studies should be presented. The summary should not describe each included study individually, but instead should summarize how the included studies vary in terms of design, number of participants, and important effect modifiers outlined in the protocol (e.g. populations and settings, interventions, comparators, outcomes or funding sources). An ‘Overview of synthesis and included studies’ (OSIS) table should be used to summarize key characteristics, and assist readers in matching studies to comparisons for synthesis (guidance on this is available in the RevMan Knowledge Base). See Chapter 9 for further guidance on summarizing study characteristics.
More details about each included study should be presented in the ‘Characteristics of included studies’ supplementary material. These are organized in tables and should include (at a minimum) the following information about each included study:
- basic study design or design features;
- baseline demographics of the study sample (e.g. age, sex/gender, key equity characteristics);
- sample size;
- details of all interventions (including what was delivered, by whom, in which setting, and how often; for more guidance see the TIDieR (Hoffmann et al 2017) and TIDieR PHP (Campbell et al 2018) reporting guidelines;
- outcomes measured (with details on how and when they were measured);
- funding source; and
- declarations of interest among the primary researchers.
Studies that may appear to some readers to meet the eligibility criteria, but which were excluded, should be listed in the ‘Characteristics of excluded studies’ supplementary material, and an explicit reason for exclusion should be provided (one reason is usually sufficient, and all reasons should be consistent with the stated eligibility criteria). It is not necessary to include every study excluded at the full text screening stage in the table; rather, authors should use their judgement to identify those studies most likely to be considered eligible by readers, and hence most useful to include here. A succinct summary of the reasons why studies were excluded from the review should be provided in the Results section.
It is helpful to make readers aware of any completed studies that have been identified as potentially eligible but have not been incorporated into the review. This may occur when there is insufficient information to determine whether the study meets the eligibility criteria of the review, or when a top-up search is run immediately prior to publication and the review authors consider it unlikely that inclusion of the study would change the review conclusions substantially. A description of such studies can be provided in the ‘Characteristics of studies awaiting classification’ supplementary material.
Readers should also be made aware of any studies that meet the eligibility criteria for the review, but which are still in progress and hence have no results available. This serves several purposes. It will help readers assess the stability of the review findings, alert research funders about ongoing research activity, help inform research implications, and can serve as a useful basis for deciding when an update of the review may be needed. A description of such studies can be provided in the ‘Characteristics of ongoing studies’ supplementary material.
III.3.5.2 Risk of bias in included studies
To help readers determine the credibility of the results of included studies, review authors should provide an overview of their risk-of-bias assessments in this section of the Results. For example, this might include overall comments on key domains that influenced the overall risk of bias judgement (e.g. the extent to which blinding was implemented across all included trials), and an indication of whether important differences in overall risk of bias were observed across outcomes. It is not necessary to describe the individual domain assessments of each included study or each result here. If risk of bias assessments were very similar (or identical) for all outcomes in the review, a summary of the assessments across studies should be presented here. If risk of bias assessments are very different for different outcomes, this section should be very brief, and summaries of the assessments across studies should be provided within the ‘Synthesis of results’ section alongside the relevant results.
If RoB 2 or ROBINS-I has been used, result-level ‘risk of bias’ tables should be included to summarize the risk of bias judgements for each domain for each study included in the synthesis. For RoB 2, these tables can be generated in RevMan, and summaries of risk of bias assessments can also be added to forest plots presenting the results of meta-analysis. Authors should use an additional supplementary material for ROBINS-I risk of bias tables. More detailed assessments, including the consensus responses to each signalling question and comments to support each response, can be made available as an additional file in a publicly available data repository.
Cochrane guidance specific to the presentation and reporting of risk of bias assessments using the RoB 2 tool is available at https://methods.cochrane.org/risk-bias-2, and for ROBINS-I at https://methods.cochrane.org/robins-i. Chapter 7, Chapter 8 and Chapter 25 present further guidance on risk of bias assessment.
III.3.5.3 Synthesis of Results
Review authors should summarize in text form the results for all pre-specified review outcomes, regardless of the statistical significance, magnitude or direction of the effects, or whether evidence was found for those outcomes. The text should present the results in a logical and systematic way. This can be done by organizing results by population or comparison (e.g. by first describing results for the comparison of drug versus placebo, then describing results for the comparison of drug A versus drug B).
If meta-analysis was possible, synthesized results should always be accompanied by a measure of statistical uncertainty, such as a 95% confidence interval. If other synthesis methods were used, authors should take care to specifically state the methods used. In particular, unless vote counting based on the direction of effect is used explicitly, authors should avoid the inadvertent use of vote counting in text (e.g. “the majority of studies found a positive effect”) (Cumpston et al 2023). It is also helpful to indicate the amount of information (numbers of studies and participants) contributing to each synthesis. If additional studies reported results that could not be included in synthesis (e.g. because results were incompletely reported or were in an incompatible format), these additional results should be reported in the review. If no data were available for particular review outcomes of interest, review authors should say so, so that all pre-specified outcomes are accounted for. Guidance on summarizing results from meta-analysis is provided in Chapter 10, from network meta-analysis in Chapter 11, and for methods other than meta-analysis in Chapter 12.
It is important that the results of the review are presented in a manner that ensures the reader can interpret the findings accurately. The direction of effect (increase or decrease, benefit or harm), should always be clear to the reader, and the minimal important difference in the outcome (if known) should be specified. Review authors should consider presenting results in formats that are easy to interpret. For example, standardized mean differences are difficult to interpret because they are in units of standard deviation, but can be re-expressed in more accessible formats (see Chapter 15).
In addition to summarizing the effects of interventions, review authors should also summarize the results of any subgroup analyses (or meta-regression), sensitivity analyses, and assessments of the risk of bias due to missing results (if performed) that are relevant to each synthesis. A common issue in reporting the results of subgroup analyses that should be avoided is the misleading emphasis placed on the intervention effects within subgroups (e.g. noting that one group has a statistically significant effect) without reference to a test for between-subgroup difference (see Chapter 10).
A ‘Summary of findings’ table is a useful means of presenting findings for the most important comparisons and outcomes, whether or not evidence is available for them. In a published Cochrane Review, all ‘Summary of findings’ tables are included before the Background section. A ‘Summary of findings’ table typically:
- includes results for one clearly defined population group;
- indicates the intervention and the comparator;
- includes seven or fewer patient-important outcomes;
- describes the characteristics of the outcomes (e.g. scale, scores, follow-up);
- indicates the number of participants and studies for each outcome;
- presents at least one estimate of the typical risk or score for participants receiving the comparator intervention for each outcome;
- summarizes the intervention effect (if appropriate), and;
- includes an assessment of the certainty of the body of evidence for each outcome.
The assessment of the certainty of the body of evidence should follow the GRADE approach, which includes considerations of risk of bias, indirectness, inconsistency, imprecision and publication bias (see Chapter 14). Where available, the GRADE assessment should always be presented alongside each result wherever it appears (for example, in the Results, Discussion or Abstract).
A common mistake to avoid is the confusion of ‘no evidence of an effect’ with ‘evidence of no effect’. When a confidence interval includes the possibility of no effect, it is wrong to claim that it shows that an intervention has no effect or is no different from the control intervention, unless the confidence interval is narrow enough to exclude a meaningful difference in either a positive or negative direction. Where confidence intervals are compatible with either a positive and negative, or positive and negligible effect, this is factored into an assessment of the imprecision of the result through GRADE. Authors can therefore report the size and direction of the central effect estimate as observed, alongside an assessment of its uncertainty.
III.3.5.4 Presenting results of studies and syntheses in tables and figures
Simple summary data for each intervention group (such as means and standard deviations), as well as estimates of effect (such as mean differences), should be presented for each study, for each outcome of interest to the review, in the Analyses supplementary material. The Analyses supplementary material has a hierarchical structure, presenting results in forest plots or other table formats, grouped first by comparison, and then for each outcome assessed within the comparison. Authors can also record in each table the source of all results presented, in particular, whether results were obtained from published literature, by correspondence, from a trials register, or from another source (e.g. clinical study report). Presenting such information facilitates attempts by others to verify or reproduce the results (Page et al 2018).
In addition to the Analyses supplementary material, review authors should include the main forest plots and tables that help the review address its objectives and support its conclusions as Figures and Tables within the main body of the review.
Forest plots display effect estimates and confidence intervals for each individual study and the meta-analysis (Lewis and Clarke 2001). Forest plots created in RevMan typically illustrate:
1. the summary statistics (e.g. number of events and sample size of each group for dichotomous outcomes) for each study;
2. point estimates and confidence intervals for each study, both in numeric and graphic format;
3. a point estimate and confidence interval for the meta-analytic effect, both in numeric and graphic format;
4. the total number of participants in the experimental and control groups;
5. labels indicating the interventions being compared and the direction of effect;
6. percentage weights assigned to each study;
7. the risk of bias in each point estimate, including the overall judgement and judgements for each domain;
8. estimates of heterogeneity (e.g. Tau2) and inconsistency (I2);
9. a statistical test for the meta-analytic effect.
For reviews using network meta-analysis, a range of figures and table formats may be appropriate to present both the network of evidence and the results of the analysis. These may include a network diagram, contribution matrix, forest plot or rankogram (see Chapter 11 for more details).
If meta-analysis was not possible or appropriate, or if the results of some studies could not be included in a meta-analysis, the results of each included study should still be presented in the review. Wherever possible, results should be presented in a consistent format (e.g. an estimate of effect such as a risk ratio or mean difference with a confidence interval, which may be calculable from the available data even if not presented in the primary study). Where meta-analysis is not used, review authors may find it useful to present the results of studies in a forest plot without calculating a meta-analytic effect.
Where appropriate, authors might consider presenting alternative figures to present the results of included studies. These may include a harvest plot, effect direction plot or albatross plot (see Chapter 12 for more details).
Figures other than forest plots and funnel plots may be produced in software other than RevMan and included as Figures in a Cochrane Review.
Review authors should ensure that all statistical results presented in the main review text are consistent between the text and tables or figures, and across all sections of the review where results are reported (e.g. the Abstract, Plain language summary, ‘Summary of findings’ tables, Results and Analyses supplementary material).
If authors wish to make additional data available, such as completed data collection forms or full datasets and code used in statistical analysis, these may be provided as additional files through a publicly available repository (such as the Open Science Framework) and cited in the review.
Authors should avoid presenting tables or forest plots for comparisons or outcomes for which there are no data (i.e. no included studies reported that outcome or comparison). Instead, authors should note in the text of the review that no data are available for the comparisons. However, if the review has a ‘Summary of findings’ table, the main outcomes should be included in this irrespective of whether data are available from the included studies.
III.3.6 Discussion
A structured discussion can help readers consider the implications of the review findings. Standard Discussion subheadings in Cochrane Reviews provide the structure for this section.
Summary of main results: It is useful to provide a concise description of results for the main outcomes of the review, but this should not simply repeat text provided elsewhere. If the review has a number of comparisons this section should focus on those that are most prominent in the review, and that address the main review objectives. Review authors should avoid repeating all the results of the synthesis, but be careful to ensure that all summary statements made in the Discussion are supported by and consistent with the results presented elsewhere in the review.
Limitations of the evidence included in the review: This section should present an assessment of how well the evidence identified in the review addressed the review question. It should indicate whether the studies identified were sufficient to address all of the objectives of the review, and whether all relevant types of participants, interventions and outcomes have been investigated. Information presented under ‘Description of studies’ will be useful to draw on in writing this part of the discussion. This section should also summarize the considerations that led to downgrading or upgrading the certainty of the evidence in their implementation of GRADE. This information can be based on explanations for downgrading decisions alongside the ‘Summary of findings’ tables in the review.
Limitations of the review process: It is important for review authors to reflect on and report any decisions they made that might have introduced bias into the review findings. For example, rather than emphasizing the comprehensiveness of the search for studies, review authors should consider which aspects of the design or execution of the search could have led to studies being missed. This might occur because of the complexity and low specificity of the search, because the indexing of studies in the area is poor, or because searches beyond bibliographic databases did not occur. If attempts to obtain relevant data were not successful, this should be stated. Additional limitations to consider include contestable decisions relating to the inclusion or exclusion of studies, synthesis of study results, or grouping of studies for the purposes of subgroup analysis. For example, review authors may have decided to exclude particular studies from a synthesis because of uncertainty about the precise details of the interventions delivered, measurement instrument used, or where it has not been possible to retrieve subgroup level data. If data were imputed and alternative approaches to achieve this could have been undertaken, this might also be acknowledged. It may be helpful to consider tools that have been designed to assess the risk of bias in systematic reviews (such as the ROBIS tool (Whiting et al 2016)) when writing this section.
Agreements and disagreements with other studies or reviews: Review authors should also discuss the extent to which the findings of the current review agree or disagree with those of other reviews. Authors could briefly summarize the conclusions of previous reviews addressing the same question, and if the conclusions contrast with their own, discuss why this may have occurred (e.g. because of differences in eligibility criteria, search methods or synthesis approach).
Further guidance on issues for consideration in the Discussion section is presented in Chapter 14 and Chapter 15.
III.3.7 Conclusions
There are two standard sections in Cochrane Reviews devoted to the authors’ conclusions.
Implications for practice: In this section, review authors should provide a general interpretation of the evidence so that it can inform healthcare or policy decisions. The implications for practice should be as practical and unambiguous as possible, should be supported by the data presented in the review and should not be based on additional data that were not systematically compiled and evaluated as part of the review. Recommendations for how interventions should be implemented and used in practice should not be given in Cochrane Reviews, as they may be inappropriate depending on the different settings and individual circumstances of readers. Authors may be helpful to readers by identifying factors that are likely to be relevant to their decision making, such as the relative value of the likely benefits and harms of the intervention, participants at different levels of risk, or resource issues. If the review considered equity, discuss the equity-related implications for practice and policy.
Implications for research: This section of a Cochrane Review is often used by people making decisions about future research, and review authors should try to write something that will be useful for this purpose. Implications for how research might be done and reported (e.g. the need for randomized trials rather than other types of study, for better descriptions of interventions, or for the routine collection of patient-important outcomes) should be distinguished from what future research should be done (e.g. research in particular subgroups of people, or an as-yet-untested experimental intervention). In addition to important gaps in the completeness and applicability of the evidence noted in the Discussion, any factors that led to downgrading the evidence as part of a GRADE assessment may provide suggestions to be addressed by future research. This could include avoidable sources of bias or larger studies. This section should also draw on what is known about any ongoing studies identified from trials register searches, and any information about ongoing or recently completed studies can be used to guide recommendations on whether new studies should be initiated. If the review considered equity, discuss the equity-related implications for research. It is important that this section is as clear and explicit as possible. General statements that contain little or no specific information, such as “Future research should be better conducted” or “More research is needed” are of little use to people making decisions, and should be avoided.
III.3.8 Additional information
A Cochrane Review should include several pieces of additional, administrative information, many of which are standard in other journals. These include acknowledgements, contributions of authors, declarations of interest, sources of support, registration and protocol details, and availability of data, code and other materials.
Acknowledgements: Review authors should acknowledge the contribution of people not listed as authors of the review and any contributions to searching, data collection, study appraisal or statistical analysis performed by people not listed as authors. Written permission is required from those listed in this section.
Contributions of authors: The contributions of each author to the review should be described. It is helpful to specify which authors were involved in each of the following tasks: conception of the review; design of the review; co-ordination of the review; search and selection of studies for inclusion in the review; collection of data for the review; assessment of the risk of bias in the included studies; analysis of data; assessment of the certainty in the body of evidence; interpretation of data, and writing of the review. Refer to the Cochrane Library editorial policy on authorship for the criteria that must be met to be listed as an author.
Declarations of interest: All authors should report any present or recent affiliations or other involvement in any organization or entity with an interest in the review’s topic that might lead to a real or perceived conflict of interest. The dates of the involvement should be reported. For reviews whose titles were registered prior to 14 October 2020, and for updates which were underway before that date, the relevant time frame for interests begins three years before the original registration of the review with Cochrane, before the beginning of an individual author’s first involvement with the review, or before the decision to commence work on a review update. For all other reviews and updates, the relevant time frame for interests begins three years before the submission of the initial draft article, or three years before the beginning of an individual author’s first involvement. If there are no known conflicts of interest, this should be stated explicitly, for example, by writing “None known”. Authors should make themselves aware of the restrictions in place on authorship of Cochrane Reviews where conflicts of interest arise. Refer to the Cochrane Library editorial policy on conflicts of interest for full details.
Sources of support: Authors should acknowledge grants that supported the review, and other forms of support, such as support from their university or institution in the form of a salary. Sources of support are divided into ‘internal’ (provided by the institutions at which the review was produced) and ‘external’ (provided by other institutions or funding agencies). Each source, its country of origin and what it supported should be provided. Authors should make themselves aware of the restrictions in place on funding of Cochrane Reviews by commercial sources where conflicts of interest may arise. Refer to the Cochrane Library editorial policy on conflicts of interest for full details.
Registration and protocol: Authors should provide the DOIs of protocols or previous versions of the review. If the systematic review is registered, authors should cite the review’s registration record number
Data, code and other materials: Cochrane requires, as a condition for publication, that the data supporting the results in systematic reviews published in the Cochrane Database of Systematic Reviews be made available for users, and that authors provide a data availability statement.
Analyses and data management are preferably conducted within Cochrane’s authoring tool, RevMan, for which computational methods are publicly available. Data entered into RevMan, such as study data, analysis data, and additional information including search results, citations of included and excluded studies, and risk of bias assessments are automatically made available for download from Cochrane Reviews published on the Cochrane Library. Scripts and artefacts used to generate analyses outside of RevMan which are presented in the review should be publicly archived and cited within the review’s data availability statement. External files, such as template data extraction forms or other data sets, can be added to a disciplinary or general repository and cited within the review. Refer to the Cochrane Library editorial policy on data sharing for full details.
III.4 Chapter information
Authors: Miranda Cumpston, Toby Lasserson, Ella Flemyng, Matthew J Page,
Acknowledgements: We thank previous chapter author Jacqueline Chandler, on whose text this version is based. This chapter builds on an earlier version of the Handbook (Version 5, Chapter 4: Guide to the contents of a Cochrane protocol and review), edited by Julian Higgins and Sally Green. We thank them for their contributions to the earlier chapter. We thank Sue Brennan, Rachel Churchill, Robin Featherstone, Ruth Foxlee, Kayleigh Kew, Nuala Livingstone and Denise Mitchell for their feedback on this chapter.
Declarations of interest: Toby Lasserson and Ella Flemyng are employees of Cochrane. Matthew Page co-led the development of the PRISMA 2020 statement.
III.5 References
Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ 2001; 322: 1479-1480.
Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology 2016; 69: 225-234.
III.S1 Supplementary material: Considerations and recommendations for figures in Cochrane reviews
Go to the III.S1 upplementary material: Considerations and recommendations for figures in Cochrane reviews page.
III.S2 Supplementary material: Guidance for writing a Cochrane Plain language summary
Go to the III.S2 Supplementary material: Guidance for writing a Cochrane Plain language summary page.
