Skip to Main Content

Systematic reviews and Meta-analysis in Business/Management

This guide is available for researchers and students undertaking systematic reviews or meta-analysis on Management or Business-related topics. It highlights best practices to improve reproducibility.

Determining your review question

Developing and refining your research question is one of the most important steps in a synthesis project. The question should be well-defined, with an appropriate and reasonable scope.  

Question frameworks can be used to help focus your research question. Two examples of frameworks relevant to management questions are presented below.

PICO

P - Population (this can be groups or categories of individuals or organizations, such as knowledge workers, post-secondary students, or SMEs)

I/E - Intervention or Exposure (this is your independent variable)

C - Comparison (if relevant)

O - Outcome of interest (this is your dependent variable)

Example question: Does gamification of training programs (I) increase motivation for compliance training (O) than non-gamified approaches (C) in banking and insurance industry employees (P)? 


CIMO

C - Context (individuals, groups of individuals, institutions, etc. Similar to population in the PICO) 

I - Intervention (event or action being studies; independent variable)

M - Mechanism (that explain, bridge, or enable the intervention to lead to the outcome)

O - Outcomes (measurable effects; dependent variable)

 

  • Harris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. Journal of clinical epidemiology, 97, 39-48. https://doi.org/10.1016/j.jclinepi.2017.10.023
  • Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative health research, 22(10), 1435-1443. https://doi.org/10.1177%2F1049732312452938
  • Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Information and Libraries Journal, 19(2), 113-115. https://doi.org/10.1046/j.1471-1842.2002.00378.x
  • Booth, A. (2006). Clear and present questions: formulating questions for evidence based practice, Library Hi Tech, 24(3), 355-368. doi: https://doi.org/10.1108/07378830610692127

Eligibility criteria

Eligibility criteria determine which studies will be included in your systematic review, and which studies will not. They may sometimes be referred to as inclusion/exclusion criteria.


1) Eligibility criteria related to PICO elements. These are additional details about each element that show where the boundaries lie and what variations are acceptable. Criteria include:

  • Population characteristics (adult, child, employee sub-group, etc.)
  • Geographic region 
  • Setting (organizational, public services, educational setting, etc)
  • Reported outcomes (self-report instruments versus objective measures)

Example

A systematic review is being conducted that looks at "what factors increase the likelihood of X in leaders". Leaders are the population (P) in the research question.

You might decide to specify further that: 1) leaders in public service organizations will be excluded (since they do not operate in a similar environment), and 2) leaders in private corporations have to be at the level of middle management or higher in order to be included. 


2) Additional criteria. These are some commonly used criteria that may be used to determine study eligibility:

  • Study design (interventional, observational, cross-sectional, case study, etc)
  • Sample size (minimum sample size for a study to be included)
  • Date of publication 
  • Language of publication
  • Data type (scholarly only, or scholarly and trade, but not dissertations, etc)
  • Type of publication (empirical studies, theoretical, editorials, etc)

Eligibility criteria should be pre-specified at the protocol development stage, before your data collection (search for studies) has been completed. Your comprehensive database search strategies may incorporate some of the specified eligibility criteria (such as publication date, language, publication type, etc)

Changing criteria after the results are known may introduce bias, and should be avoided. If you change your criteria at a later stage, you may need to revisit your database search strategies, as they may be affected.

Additionally, you must report (in your final manuscript) any changes that were made in a effort to be transparent, and provide justification for the changes.

  • Watson, D., Tregaskis, O., Gedikli, C., Vaughn, O., & Semkina, A. (2018). Well-being through learning: A systematic review of learning interventions in the workplace and their impact on well-being. European Journal of Work and Organizational Psychology, 27(2), 247-268. https://doi.org/10.1080/1359432X.2018.1435529

An example of a published article with detailed eligibility criteria. This article provides a very detailed description of their eligibility criteria and list them as they relate to each of the PICOS elements.

Creating a review protocol

The review protocol

A primary study would never be conducted without a research plan; similarly, evidence synthesis should not be undertaken without a research protocol. A protocol describes the planned methodology and intended process for the various steps of your review, including:

  • The research question
  • Data collection
  • Eligibility criteria and study selection process (how you will select studies for inclusion)
  • Data extraction
  • Critical appraisal 
  • Synthesis and/or meta-analysis

PRISMA - P - This checklist describes the items and level of detail that needs to be reported in a protocol for a systematic review. 


Registering/publishing your protocol

It is a good idea to have your protocol done before you begin collecting data for your review. This shows that you did not hypothesize after you knew the direction of the effect or what the results would show, and in this way reduces the opportunity for reporting bias. In some disciplines, journals may require proof that your protocol was pre-registered. 

Organizations such as Cochrane, Campbell Collaboration, JBI and others that conduct evidence synthesis publish their own protocols. However you can also upload a copy of your protocol to an open access registry or even your institutional repository.

Open Science Framework - You can pre-register your protocol on the Open Science Framework registry. It can be submitted for blinded peer-review, or can be embargoed for a set period of time (maximum 4 years).

PROSPERO - This is an international database of prospectively registered systematic reviews. Their disciplinary coverage includes: health, social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome.

Example of a systematic review protocol published by the Business & Management group at the Campbell Collaboration. 

  • Rousseau, DM., Beck, D., Kim, B., Splenda, R., Young, S. (2019) PROTOCOL: Does executive compensation predict publicly traded firms’ financial performance or inaccurate financial reporting? Campbell Systematic Reviews, 15, e1064. https://doi.org/10.1002/cl2.1064

Data management

Similar to other research projects, you should make a plan for how you will manage the vast amounts of data acquired during the course of your review project. Some of the types of data you will acquire/create are:

  • Exported files from each database
  • Citation libraries (in your citation management software)
  • Article PDFs
  • Data extraction files (Excel, Word docs)
  • Notes (documenting each step of the process)

Data management tips

  1. It is good to have a central backup of all data from your project, regardless of which member of the review team is responsible for acquiring or creating that particular type of data. This ensures that the team can always access a copy of all data, in the case of a technology accident (laptop crash, lost USB) or if someone needs to be away for an extended period of time or a personnel change.
  2. Name your files appropriately. It is good to think about a naming convention for the files themselves, as well as the folders in which you will organize your backups. Remember that 12 months from now, you may not be able to remember what 1.ris means.
    • You may want to use a multi part name which includes aspects such as: the project name, step of the review process, database name, individuals name, date, etc.
    • For naming files exported from a database during the search stage, you might name your file ProjectName_NameofDatabase_DateofSearch_#ofRecords.
    • For naming the files that were used to pilot the screening criteria, you might name your file ProjectName_PilotScreening_#ofRecords_Date_InitialsofScreener
    • For naming each of the full-text article PDFs, you might consider naming your files  FirstAuthor_Year_FirstFourWordsOfTitle