The importance of peer-review for your search strategy

Jan 2, 2019

Written by Neil Webb, Head of Systematic Review

The search strategy is the foundation of a well-conducted systematic literature review (SLR) and is the fundamental element that can affect the overall quality of the results of the final review (1, 2). The aim of a search string is to achieve a balance between sensitivity (capturing all relevant articles) and precision (eliminating irrelevant hits); a lack of balance has implications on the quality of the evidence identified and the resources required to conduct the review. In a recent study, a random sample of 70 new Cochrane SLRs published in 2015 were evaluated for the design and reporting of search strategies using recommended checklists. A problem in the design of the search was found in 73% (95% CI, 60-84%) of the reviews, and 53% (95% CI, 38-69%) of those problems could limit both the sensitivity and precision of the search strategies (3).

In addition, search string errors can result in the complete omission of highly relevant studies. Sampson et al 2006 found 82.5% of 105 MEDLINE search strings examined for errors contained an error that could potentially lower recall of relevant studies (2).

In order to enhance the quality of database search strings, peer-review feedback can be utilised. Peer-review of the search strategy provides a subjective validation, which can reduce the risk of errors, reduce the risk of missing relevant studies, and improve the balance between sensitivity and precision. The development of the PRESS checklist, an evidence based guideline in the form of a checklist, provides clear guidance for peer-reviewers to follow and ensures information specialists can check search strategies in a more formal, structured way (4). During development of the PRESS checklist, the following seven elements of the search strategy were highlighted as important to check when conducting a peer-review (2):

  • accurate translation of the research question into search concepts
  • correct choice of Boolean operators
  • accurate line numbers and absence of spelling errors
  • an appropriate text word search
  • inclusion of relevant subject headings
  • correct use of limits and filters
  • search strategy adaptations.

The importance of peer-review of search strings has been reported in three recent publications. In a study conducted by the Agency for Healthcare Research and Quality (AHRQ), which assessed the use of the PRESS checklist, the authors found that it “seems to cut down the time needed to do the review, increase response, and do a better job of identifying actual errors in search strategies” (5). In an internal investigation conducted by the Canadian Agency for Drugs and Technologies in Health (CADTH), the authors found that both the number and quality of the relevant articles retrieved in CADTH Rapid Responses reports were improved when using peer-review of search strings compared with no peer-review (6, 7).

At Source Health Economics (Source), to ensure our developed search strings for health technology assessment (HTA) submissions are comprehensive and of the highest quality, we incorporate external peer-review of search strings into all of the HTA SLRs we complete. As a result of Source’s strategic partnership with the School of Health and Related Research (ScHARR), Sheffield University, peer-review of our developed search strings is conducted by an information specialist at ScHARR, using the PRESS checklist (4).

References
  1. Lefebvre C, Duffy S. Peer reviewing search strategies. HTAi vortal 2018. Available at: http://vortal.htai.org/index.php?q=book/export/html/918.
  2. Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006;59(10):1057-63.
  3. Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Res Synth Methods. 2018;9(3):408-16.
  4. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-6.
  5. Relevo R, Paynter R. Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.) AHRQ Publication No. 12-EHC068-EF. Rockville, MD: Agency for HealthcareResearch and Quality. Accessed via: www.effectivehealthcare.ahrq.gov/reports/final.cfm. 2012.
  6. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at Canadian Health Libraries Association (CHLA) Conference; 22-25 May 2013; Saskatoon, Saskatchewan: Canada. Available at: https://www.cadth.ca/media/is/CSpry_Poster.pdf.
  7. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at the 21st Cochrane Colloquium; 19-23 Sep 2013; Quebec: Canada. Available at: https://abstracts.cochrane.org/2013-qu%C3%A9bec-city/peer-review-literature-search-strategies-does-it-make-difference.

 

At Source Health Economics, we are committed to delivering the highest quality search strings for HTA submissions. We’re proud to announce our partnership with ScHARR, where an information specialist reviews our developed search strings using the PRESS checklist.


More Insights

Language restrictions in systematic literature reviews: a reflection on the current tone

Language restrictions in systematic literature reviews: a reflection on the current tone

Written by Alys Ridsdale, Associate Systematic Review Analyst   Introduction There are over 7,100 spoken languages across the globe (1). Despite this linguistic diversity, including language restrictions in a systematic literature ... Read more

What are the implications of joint clinical assessment (JCA) on systematic reviews?

What are the implications of joint clinical assessment (JCA) on systematic reviews?

This article offers a overview of the challenges and strategies for navigating the upcoming EU JCA process, focusing on managing PICOS criteria and deadlines in clinical systematic literature reviews (SLRs). Read more

The European Union Joint Clinical Assessment (EU JCA) – an overview

The European Union Joint Clinical Assessment (EU JCA) – an overview

This article offers a brief overview of the EU JCA process and is the first in a series of articles where we will discuss the new process and implications for HTA developers. Read more

Pros and cons of crowdsourcing for systematic review

Pros and cons of crowdsourcing for systematic review

Written by Ciara Thomas, Vicky Crowe, & David Pritchett   Introduction To achieve optimal patient outcomes, systematic literature reviews (SLRs) must synthesise high quality, contemporary evidence to inform health policy and ... Read more