Skip to Main Content

Libraries & Cultural Resources

Research Guides

Systematic reviews and Meta-analysis in Business/Management

This guide is available for researchers and students undertaking systematic reviews or meta-analysis on Management or Business-related topics. It highlights best practices to improve reproducibility.

Study selection process (screening)

Study selection (also known as screening) is the process by which records identified in the previous step are assessed for inclusion. Objectively and systematically applying the pre-specified eligibility criteria to each record in order to determine whether the article should be included helps avoid selection bias.

The first phase of study selection involves screening at the title/abstract level.

The objective of this step is to remove obviously irrelevant records.  In order to further reduce user bias, it is recommended that this process be done in duplicate (independently by two individuals).

  1. Each title/abstract is screened against the eligibility criteria independently by each reviewer and assigned a decision.
    • Yes (the title/abstract looks relevant)
    • Maybe (the title/abstract may be relevant; there is not enough detail to confirm either way)
    • No (the title/abstract is clearly not relevant)
  2. The decisions are merged or compared.
  3. Any discrepancies can be resolved by either: a) discussion and consensus, or b) by a third independent reviewer.

Records with a yes or maybe decision move on to the next stage of screening.

The second (final) phase of study selection involves screening at the full-text level.

The objective of this step is to assess each article and include only those studies that meet the pre-specified eligibility criteria. In order to reduce bias, it is recommended that this process be done in duplicate (independently by two individuals).

  1. Each full-text article is screened against the pre-specified eligibility criteria independently by each reviewer and assigned a decision.
    • Yes (the study meets all inclusion criteria and should therefore be included)
    • No (the study does not meet all inclusion criteria, or meets even one of the exclusion criteria and should therefore be excluded). At this screening stage, the reason for exclusion must be recorded, as it will need to be reported.
  2. The decisions are merged or compared.
  3. Any discrepancies can be resolved by either: a) discussion and consensus, or b) by a third independent reviewer.
  • Frampton, G. K., Livoreil, B., & Petrokofsky, G. (2017). Eligibility screening in evidence synthesis of environmental management topics. Environmental Evidence, 6(1), 27. https://doi.org/10.1186/s13750-017-0102-2
  • Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019). Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Research Synthesis Methods, 10(3), 330-342.  
  • Stoll, C. R., Izadi, S., Fowler, S., Green, P., Suls, J., & Colditz, G. A. (2019). The value of a second reviewer for study selection in systematic reviews. Research synthesis methods, 10(4), 539-545.  
  • Gartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020). Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology.  https://doi.org/10.1016/j.jclinepi.2020.01.005
  • Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC medical research methodology, 19(1), 132. https://doi.org/10.1186/s12874-019-0782-0

Tools and software for screening

Covidence is a web-based tool that helps facilitate various steps of the systematic review process.  It can be used for:

1. Deduplication of records from multiple database searches. Files can be imported in RIS or XML format, directly from a database or from a citation management tool (such as EndNote, Mendeley, Zotero, etc). Covidence de-duplicates based on the title, year, volume, and author fields.

2. Study selection. Screening is done independently, such that the 2 individual screeners cannot see how the other screened until after a record has both votes. Settings can be altered to allow single screener decisions. The tool simplifies the process of comparing decisions and automatically highlights conflicts that need to be resolved. PDFs can be uploaded and are stored in Covidence for full-text screening. It can also create a PRISMA flow diagram for you.

A number of quick help articles and videos are available

Importing references

Title and abstract screening

Resolving conflicts

Importing PDFs in bulk

Full-text screening

University of Calgary Institutional Subscription

All University of Calgary faculty, students and staff have access to Covidence through the institutional subscription. This allows you to create an unlimited number of reviews in Covidence, and also invite other authors (from any institution) to participate in these reviews.

Setting up a University of Calgary Account

New Users

  1. Request an invitation to access Covidence through the University of Calgary's institutional  subscription: https://www.covidence.org/organizations/Q4VXa/signup. Note: you must have a valid University of Calgary email address to make this request.
  2. You will receive an email acknowledgement, from which you can create a personal Covidence account that is linked to the University of Calgary's institutional subscription.

Existing Users

If you previously had a Covidence account that was created using your University of Calgary email address, you should already have access to the University of Calgary's institutional subscription in Covidence.

  1. Sign in to Covidence with your existing login information
  2. Click Start a New Review
  3. Select the University of Calgary Covidence account

Rayyan - This is a free tool that is available for screening. It is also available as a mobile app, which allows offline screening, and uploading of decisions the next time the device is connected to the internet. 

Excel - An alternative to software/tool-based screening is to use Excel. Records can be exported in CSV format directly from databases, or from a citation management tool (such as EndNote).

Here is a list of scholarly articles that compare or evaluate the performance of various screening tools:

1. Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020). Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Medical Research Methodology, 20(1), 7. https://doi.org/10.1186/s12874-020-0897-3

This article reviews 6 tools: Abstrackr, Colandr, Covidence, DRAGON, EPPI-Reviewer and Rayyan

2. Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E. (2019). Usability and acceptability of four systematic review automation software packages: a mixed method design. Systematic reviews, 8(1), 145.https://doi.org/10.1186/s13643-019-1069-6

This article review 4 tools: Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst

3. Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan—a web and mobile app for systematic reviews. Systematic reviews, 5(1), 210. https://doi.org/10.1186/s13643-016-0384-4

Libraries & Cultural Resources