Systematic reviews are a type of research methodology that involves a comprehensive and unbiased synthesis of all available evidence on a particular research question or topic. They are a rigorous and standardised approach to reviewing and analysing literature, designed to minimise bias and maximise the validity and reliability of the findings. Systematic reviews are widely used in healthcare, social sciences, education, and other fields to inform policy and practice, and are considered the gold standard for evidence-based decision-making.
Overall, systematic reviews play a critical role in advancing our understanding of complex issues and improving the quality of decision-making in several fields of practice.
1.1 Our role in supporting systematic reviews
We recommend reading the following information about this particular type of research design before contacting us.
Staff, researchers and postgraduate research students who are conducting a systematic review can contact us at [email protected] to consult with a member of the team. We will help you understand the process, provide insight into developing a search strategy and also advise on selecting bibliographic databases and other resources for a comprehensive literature search.
1.2 What is a systematic review?
“A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making.” Source: Cochrane Library (2019). About Cochrane Reviews | Cochrane Library. [online] Cochranelibrary.com. Available at: https://www.cochranelibrary.com/about/about-cochrane-reviews.
Systematic reviews primarily aim to facilitate well-informed decision-making with minimal bias by enhancing the quality of evidence. The need for systematic reviews has emerged because for most empirical questions the relevant literature is extensive, and clinicians and policymakers cannot spare the time nor have the skills to retrieve all the pertinent literature. They also cannot afford to appraise and synthesise every piece of evidence to identify the most suitable literature which will be “the all-things-considered” answer to a research question.
Without systematic reviews, decisions are likely to be based on a subset of publications which may not be representative of the whole literature, and there will be a risk that the reviewers will consciously or unconsciously select those publications which best support their views. Systematic reviews minimise bias by using techniques and presenting the literature synthesis in a format accessible to decision-makers. Unsurprisingly, systematic reviews are used increasingly to make clinical decisions, write clinical guidelines, and set research agendas.
Source: SOFAER, N. and STRECH, D. (2011). THE NEED FOR SYSTEMATIC REVIEWS OF REASONS. Bioethics, 26(6), pp.315–328. doi:10.1111/j.1467-8519.2011.01858.x. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3458717/#:~:text=The%20primary%20purpose%20of%20the,the%20relevant%20literature%20is%20extensive
1.4 Some other types of reviews
Scoping review
This type of review delivers an initial assessment of the potential size and scope of available research literature. It focuses on identifying the nature and extent of research evidence (usually including ongoing research).
Rapid review or Rapid evidence assessments
An assessment of what is already known about a policy or practice by using systematic review methods to search, locate and critically appraise existing research.
Meta-analysis
A special technique that statistically combines the results of quantitative studies to provide a more precise effect on the results.
Mixed methods/mixed studies
Refers to any combination of methods where one significant component is a literature review (usually systematic). Within a review context, it refers to a combination of review approaches. For example, combining quantitative with qualitative research or outcome with process studies.
Source: Grant, M.J. and Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), pp.91–108.
Not sure if you should conduct a systematic review or a scoping review? This article could help your decision-making.
Write and Register Protocol (note to students: please check with your supervisor for registration)
Run Search
Select Eligible Studies
Assess Quality of Eligible Studies
Extract Data
Synthesise Results
Interpret and Report Findings
1.6 Is there already a systematic review on your topic?
To see if a review on your topic is currently in progress or has already been published by someone else, use the resources listed below. If a review already exists, check its currency and quality. Does it need updating or is it of poor quality? If so, consider whether a new systematic review should be performed.
A systematic review protocol is a critical component of conducting a rigorous and transparent systematic review. It outlines the methods and procedures that will be followed in the review, ensuring that the process is systematic, transparent, and replicable.
2.1 What is a protocol?
A systematic review protocol describes the review’s rationale, hypothesis, and planned methods. It should be prepared before a review is started and used as a guide to carry out the review. The protocol should be registered and publicly available (note to students: please check with your supervisor for registration).
2.2 Registering your protocol and why is it important
Detailed protocols should be developed theoretically, made available in the public domain, and registered in a registry such as PROSPERO.
Registration provides transparency in the review process. It helps counter publication bias by providing a permanent record of prospectively registered reviews, irrespective of whether they are eventually published or not. It helps safeguard against reporting biases by revealing any differences between the methods or outcomes reported in the published review and those planned in the registered protocol. This should improve quality and increase confidence that policy or practice informed by the findings of a systematic review is drawing on best-quality evidence.
Registration allows those commissioning or planning reviews to identify whether there are any reviews already underway that address their topic of interest. This helps avoid unintended and economically wasteful duplication of effort.
2.3 How to Register a Systematic Review in PROSPERO
PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health-related outcome. Key features from the review protocol are recorded and maintained as a permanent record. PROSPERO aims to provide a comprehensive listing of systematic reviews registered at inception to help avoid duplication and reduce opportunities for reporting bias by enabling comparison of the completed review with what was planned in the protocol.
PROSPERO is produced by CRD and funded by the National Institute for Health Research (NIHR).
Allocate a few hours for the registration process. Registering your protocol is free of charge, editing it is allowed, however PROSPERO system keeps a record of all those changes.
2.4 Guidance on writing your protocol
Your protocol is a proposal of the review you are planning to perform and describes how you are going to go do it.
Note to students: for exact protocol requirements please check with your supervisor.
3. Literature Searching
Literature searching is the process of identifying, selecting, and retrieving relevant literature to support clinical research activities. It involves using a variety of search strategies and databases to identify relevant studies. The aim of literature searching is to provide a comprehensive and up-to-date overview of the available evidence. Literature searching is a critical component of clinical research, as it ensures that research questions are informed by the best available evidence and that study designs and interventions are grounded in existing knowledge.
3.1 Search Strategy
our literature search will need to be more in-depth than prior literature searches you may have done to meet the requirements for systematic reviews. You will need to think of several synonyms for your search phrases and double-check that you are searching in each database’s title, abstract, and MeSH term fields. To ensure your search is as thorough and effective as possible, make sure you are correctly using the Boolean operators AND, OR, and NOT as well as another search syntax such as truncation and wildcards.
Formulating your search question using PICO:
Most systematic review search strategies use PICO to frame their search.
First, you need to formulate a clear question. This should generate search terms. There are models which may help you do this such as PICO:
P = POPULATION / PROBLEM (e.g., patient group, problem, condition, disease, gender, age etc.). – What’s the problem? Who’s affected?
I = INTERVENTION/EXPOSURE (e.g., treatment, exposure, diagnostic test, drug, procedure) – what is being done?
C = COMPARISON (not always applicable) – what’s the alternative, comparing your intervention with another treatment or test?
O = OUTCOME (e.g., reduced mortality, improved quality of life, length of stay, cost-effectiveness, complications). – What effect will it have? You may want to include a separate search for a setting if this is relevant, e.g., critical care, hospital, primary care, etc. If you want a particular study type. e.g., RCT, this can be added as a limit at the end of your search.
A question such as ‘Does the flu vaccine help reduce absenteeism in healthcare workers? could be broken down as follows:
P
I
C
O
Health personnel Healthcare workers Healthcare staff Nurses Doctors Hospital staff etc.
Flu vaccine Influenza vaccination Influenza immunization etc.
Absence Sick leave Absenteeism etc.
It is not necessary to fill each column for the PICO model. If you have search terms in the first two columns, you can run a search.
The main thing to remember is you MUST break your question down into different concepts.
3.2 Other search frameworks
While PICO (Population, Intervention, Comparison, Outcome) is a commonly used framework for formulating research questions in systematic reviews, there are several other frameworks that can be used as alternatives or supplements. Here are a few additional question frameworks:
SPICE: This framework is commonly used in qualitative systematic reviews and stands for Setting, Perspective, Intervention/Phenomenon of Interest, Comparison, and Evaluation. It helps researchers focus on the contextual factors, perspectives, and experiences related to the phenomenon under investigation.
ECLIPSE: This framework is specifically designed for qualitative evidence synthesis and stands for Experts, Context, Location, Impact, Phenomenon of Interest, Setting, and Experience. It emphasizes capturing diverse perspectives and experiences related to the research question.
PICOT: This framework is similar to PICO but includes an additional element, Time. It stands for Population, Intervention, Comparison, Outcome, and Time. It is commonly used in healthcare research to account for the temporal dimension of the research question.
SPIDER: This framework is widely used for formulating research questions in qualitative systematic reviews and stands for Sample, Phenomenon of Interest, Design, Evaluation, and Research type. It helps researchers articulate the key elements of the question while considering the qualitative research design.
FAME: This framework is used specifically for research questions in evidence-based medicine and stands for Format, Action, Comparison, and Outcome. It focuses on intervention-based questions in clinical practice.
FINER: This framework is commonly used for research questions related to feasibility and stands for Feasible, Interesting, Novel, Ethical, and Relevant. It helps researchers assess the practicality and relevance of a research question.
Remember that these frameworks serve as guides and can be adapted based on the specific requirements of your systematic review or research study. Choosing a framework that aligns with your research objectives and allows for a clear and structured formulation of your question is essential.
Booth A, Noyes J, Flemming K, Moore G, Tunçalp Ö, Shakibazadeh E. Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Glob Health. 2019 Jan 25;4(Suppl 1):e001107. doi: 10.1136/bmjgh-2018-001107. PMID: 30775019; PMCID: PMC6350737.
3.3 Understanding search strategies
Systematic review search strategies are more complex than those you are perhaps used to.
They frequently include many lines of search terms, a mixture of MeSH and free text terms, translating search syntax from one database to another, and use of advanced search syntax such as truncation (using *), proximity and wildcards. Please look at this guide by the University of Exeter Library for a detailed explanation of how to ‘decode’ search strategies in systematic reviews.
3.4 Where to conduct your literature search
Electronic databases (embed URL for each database)
More sources: Consult your course subject resources area for more search platforms.
3.5 Is PubMed appropriate for systematic reviews?
Although most of the usual medical and healthcare databases are appropriate for use in systematic reviews, PubMed should be avoided and here is why:
PubMed lacks proximity operators search functionality, making it difficult to search consistently across several databases.
The major problem is that PubMed searches are not reproducible and transparent because PubMed uses machine learning algorithms working behind the scenes which are invisible to the searcher making transparency and reproducibility impossible which are both of key importance in scientific reporting and experiments. Without these present in the search strategy, a systematic review falls at the first hurdle when being critically appraised.
PubMed issue with PRISMA-S checklist was launched recently, outlining all the reporting requirements for literature searching in systematic reviews. On the PRISMA checklist item, 8 requires you to “include the search strategies for each database and information source, copied and pasted exactly as run”. Note ‘exactly as run’. This is not possible in PubMed, therefore a Medline search via EBSCO or another platform is suggested.
Hand searching involves scanning the content of pre-selected peer-reviewed biomedical journals, conference proceedings and abstracts. It is an important way of identifying very recent publications that have not yet been indexed by electronic bibliographic databases. Hand searching can also ensure the inclusion of all types of evidence including letters or commentaries, which may not be indexed by databases and therefore not captured as literature search results and/or remedy poor or inaccurate database indexing that can result in even the most carefully constructed search strategy, failing to identify those relevant studies. Choosing which journals to hand search can be achieved by analysing the results of the database searches to identify the journals that contain the largest number of relevant studies.
3.7 Grey literature
The term grey literature refers to information produced outside of conventional scholarly publication and dissemination routes and is frequently underrepresented in bibliographic databases; these include PhD theses, conference proceedings, reports, unpublished trial data, and more.
You may risk bias to your results if you leave out grey literature.
CADTH is an independent, not-for-profit organisation responsible for providing healthcare decision-makers with objective evidence to help make informed decisions about the optimal use of health technologies, including: drugsdiagnostic testsmedical, dental, and surgical devices and procedures. In addition to evidence, they also provide advice, recommendations, and tools.
The International Health Technology Assessment Database has been relaunched and provides free access to bibliographic information about ongoing and published health technology assessments commissioned or undertaken by HTA organisations from around the world.
Mednar is a free, medically-focused deep web search engine that uses Explorit Everywhere, an advanced search technology by Deep Web Technologies. As an alternative to Google, Mednar accelerates your research with a search of authoritative public and deep web resources, returning the most relevant results to one easily navigable page
The National Academies Press has more than 10,000 books downloadable for free (registration required), produced by the National Academies of Sciences, Engineering, and Medicine.
The Nuffield Trust is an independent health charity. Their aim is to improve the quality of health care in the UK by providing evidence-based research and policy analysis, and informing/generating debate.
OAIster is a union catalogue of millions of records that represent open-access resources, built through harvesting from collections worldwide. Includes more than 50 million records that represent digital resources from more than 2,000 contributors.
OECD iLibrary is the online library of the Organisation for Economic Cooperation and Development (OECD) featuring its books, papers and statistics and is the knowledge base of OECD’s analysis and data.
OpenGrey covers Science, Technology, Biomedical Science, Economics, Social Science and Humanities in Europe.Includes:Open access to 700.000 bibliographical references, with the facility to export records and locate the documents. Technical or research reports, doctoral dissertations, some conference papers, some official publications, and other types of grey literature. The site includes preprints from the GL conferences (GreyNet International) in full text.
RAND produce reports on a range of public health topics, freely available as eBooks. You can search over 20,000 RAND publications dating back to 1946, and filter by theme.
Trip is a clinical search engine, but as well as research evidence, it also provides a gateway to other content types, including images, videos, patient information leaflets, educational courses and news. The Pro version is now funded for NHS and Public Health staff. For Local Authority PH teams please activate IP address linkage by emailing [email protected]
3.8 Snowballing
After screening and including the articles which meet your inclusion criteria, it may be appropriate to ‘snowball’ your search involving screening the articles that cite your included papers.
You can search for the titles of each included article in Scopus and any listed citing article that meets your inclusion criteria should also be included in the review.
Note: Included studies found this way should be detailed in PRISMA diagram
3.9 Pearling (secondary searching)
After you have reviewed all the papers found in databases, including grey literature and decided which ones will be included in the review you can look through their reference lists for any articles in their reference lists which meet your inclusion criteria, those need to be included in your systematic review also.
Note: Included studies found this way should be detailed in PRISMA diagram
4. Managing your search results
Managing literature search results in research refers to the process of organising, storing, and tracking the publications that have been identified and reviewed during a literature search. This may involve using software tools or databases to manage citations and abstracts, as well as keeping track of full-text articles that have been obtained. Effective management of literature search results can help researchers to avoid duplication of effort and it is a critical component of the research process and is essential for ensuring the quality and credibility of the process.
4.1 Saving searches
It is good practice to keep a record of what your search history (what you searched), and when.
This will save time if you have a similar enquiry or need to re-run the search in future.
It also shows the steps you took to find your information (for transparency).
Most platforms allow you to save your search history permanently.
4.2 De-duplicating searches
To remove the duplicates from your results you will need to use a reference manager such as RefWorks or EndNote which will perform the deduplication for you automatically.
4.3 Reference Management Software
At EHU you have access to RefWorks and EndNote.
The library can provide you with training for both software as well as two alternatives – Mendeley and Zotero.
5. How to document your searches
How to document your searches: This stage is a part of reporting standard for systematic reviews which is the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. It provides a checklist of items that should be included in a systematic review report to ensure that it is transparent and complete. The PRISMA statement is widely used and recognised as the standard for reporting systematic reviews in many fields, including healthcare and social sciences.
5.1 Documenting literature searches
Documenting database searches in a systematic review is important for transparency, reproducibility, and minimising bias. It allows others to replicate the search process, verify the accuracy of results and reduces the risk of selective reporting. Thorough documentation enhances the reliability and integrity of the review and supports evidence-informed decision-making.
To document the databases searched in a systematic review:
Keep a detailed record of all databases and other sources searched.
List the names of the databases searched.
Record the search strategies used for each database, including search terms, operators, and limits.
Note the date of each search.
Include the complete search strings if applicable.
Document any additional sources searched, such as reference lists or experts.
Update the search log as the review progresses.
Archive the search records for future reference and transparency.
5.2 Reporting Standards
PRISMA (the Preferred Reporting Items for Systematic Reviews and Meta-analyses) is ‘an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.’ The PRISMA checklist is a useful guideline of content that should be reported and included in the final write-up of your systematic review and will help when in the planning stages as well.
PRISMA-S is the reporting guideline for systematic review search methods.
Although PRISMA is widely used as a reporting guideline for systematic reviews in the UK, Canada and other countries, some researchers may choose to use other reporting guidelines depending on their specific research question and field of study.
In addition to PRISMA, some other reporting guidelines that are commonly used for systematic reviews include:
Cochrane Handbook for Systematic Reviews of Interventions: This is a comprehensive resource for conducting systematic reviews and meta-analyses of healthcare interventions, developed by the Cochrane Collaboration. Cochrane Handbooks can be found here. Please choose the appropriate link for your review type.
JBI Manual for Evidence Synthesis: The Joanna Briggs Institute (JBI) provides a variety of resources and tools for conducting systematic reviews in healthcare and social sciences, including their Manual for Evidence Synthesis.
ENTREQ (Enhancing Transparency in Reporting the Synthesis of Qualitative Research) Guidelines from the EQUATOR Network: These guidelines were developed to improve the transparency and reporting of systematic reviews of qualitative research.
It’s worth noting that there are many other reporting guidelines available for systematic reviews, and the choice of which to use may depend on factors such as the research question, study design, and field of study.
6. Screening
Once you have completed your literature search, it is time to start screening the results to remove the papers that do not meet the inclusion criteria stated in your protocol. It is recommended to have two reviewers at this stage and a team at the end of screening to resolve any conflict.
6.1 Title and Abstract screening
During this step, the review team needs to perform the laborious but essential first step of synthesis of the existing literature during which you will look to see if any papers meet the criteria for inclusion outlined in your protocol. Access this journal article which provides a practical set of best practice guidelines to help you carry out this essential step:
To obtain the articles you wish to include for full-text screening, if immediate access is not available to you via the EHU library subscriptions, you will need to ask us to provide you with a copy by using this link: You want it, we get it.
6.3 Full-text screening
The full text screening stage is a critical step in conducting a systematic review which follows “Title and abstract screening” step. It involves evaluating the full-text articles or documents that have been identified as potentially relevant during the initial screening phase. The purpose of this stage is to determine whether each article meets the predetermined inclusion criteria for the review.
To help you with this step you may utilise systematic review screening software such as Rayyan to help you screen your systematic review.
Rayyan is a web-based platform designed to streamline the screening process by facilitating reviewer collaboration. It allows for easy uploading and sharing of studies, enabling multiple reviewers to independently assess and categorise them based on inclusion or exclusion criteria. Rayyan also offers features such as automated deduplication, real-time progress tracking, and customisable screening forms. By utilising Rayyan, students can enhance the efficiency and accuracy of their systematic review screening process, ultimately saving time and improving the quality of their research.
Critical appraisal in systematic reviews involves rigorous evaluation and synthesis of primary research studies to assess their quality, relevance, and potential biases, ensuring that the overall conclusions drawn are reliable and valid. It plays a vital role in establishing the credibility and trustworthiness of the systematic review findings.
7.1 Quality assessment tools
For this step, you will need to assess the quality of the included citations and determine whether they are relevant, reliable, valid, or applicable.
Please explore the following 8 bite-sized critical appraisal programme modules by the NHS Knowledge For Healthcare Academy, designed to support you with understanding the different methods and tools to carry out critical appraisal
Introduction to critical appraisal.
Introduction to equity considerations in critical appraisal.
Introduction to critical appraisal of randomised controlled trials.
Introduction to interpreting results for critical appraisal.
Introduction to critical appraisal of systematic reviews.
Introduction to critical appraisal of qualitative studies.
Introduction to critical appraisal of diagnostic studies.
Introduction to critical appraisal tools.
For assessing the quality of the papers that meet the inclusion criteria you can choose a critical appraisal tool that are essentially questionnaires created to assist you to perform this process.
Gathering pertinent information from studies that you have determined are eligible to be included in your review and structuring it in a way that will enable you to synthesise the research and reach conclusions is known as data extraction. This process needs to be done according to PRISMA or Cochrane (https://opal.latrobe.edu.au/articles/journal_contribution/Data_extraction_template/6818852) reporting checklists and guidelines.
After data extraction, you will provide a deeper understanding of the selected studies. A qualitative summary should be included in every evaluation, however, they may or may not be a meta-analysis.
9.1 Qualitative Synthesis
A narrative of studying, evaluating, and synthesising the body of information in your review is known as a qualitative synthesis. All systematic reviews must include this type of synthesis.
In this section:
Give a broad overview of the characteristics and conclusions of the studies that you included.
Analyse the connections between research, looking for trends and diversity.
discuss how the evidence collected relates to the review’s central question.
If a meta-analysis was performed, you will need to describe, interpret and evaluate how reliable the results were.
Discuss the advantages and disadvantages of the collected evidence as a whole, taking into account an overall evaluation of the risk of bias among them.
Comment about any evidence gaps, such as patient groups that haven’t been properly researched or whose outcomes may vary.
When applicable, compare the review’s conclusions with the prevailing opinion regarding the review’s question.
A quantitative synthesis, also known as a meta-analysis, combines and examines the findings of several research studies using statistical methods. A systematic review that combines studies, such as Randomised Controlled Trials (RCTs), with other types of research design should include meta-analyses. A meta-analysis is not present in all systematic reviews. Please refer to the Cochrane handbook for more information:
Note: it is recommended to consult a statistician at this stage.
10. Writing the manuscript
When writing up, keep the requirements of the funding body or your target journal in mind.
There are several guidelines available that you can refer to when preparing your manuscript. In any case, you can harvest most of the introduction and methodology section from your protocol.
Support for staff and postgraduate research students:
Training and support for literature searching and general consultation for writing a protocol is available from Academic Engagement team, please email: [email protected].
