The importance of peer-review for your search strategy

Jan 2, 2019

Written by Neil Webb, Head of Systematic Review

The search strategy is the foundation of a well-conducted systematic literature review (SLR) and is the fundamental element that can affect the overall quality of the results of the final review (1, 2). The aim of a search string is to achieve a balance between sensitivity (capturing all relevant articles) and precision (eliminating irrelevant hits); a lack of balance has implications on the quality of the evidence identified and the resources required to conduct the review. In a recent study, a random sample of 70 new Cochrane SLRs published in 2015 were evaluated for the design and reporting of search strategies using recommended checklists. A problem in the design of the search was found in 73% (95% CI, 60-84%) of the reviews, and 53% (95% CI, 38-69%) of those problems could limit both the sensitivity and precision of the search strategies (3).

In addition, search string errors can result in the complete omission of highly relevant studies. Sampson et al 2006 found 82.5% of 105 MEDLINE search strings examined for errors contained an error that could potentially lower recall of relevant studies (2).

In order to enhance the quality of database search strings, peer-review feedback can be utilised. Peer-review of the search strategy provides a subjective validation, which can reduce the risk of errors, reduce the risk of missing relevant studies, and improve the balance between sensitivity and precision. The development of the PRESS checklist, an evidence based guideline in the form of a checklist, provides clear guidance for peer-reviewers to follow and ensures information specialists can check search strategies in a more formal, structured way (4). During development of the PRESS checklist, the following seven elements of the search strategy were highlighted as important to check when conducting a peer-review (2):

  • accurate translation of the research question into search concepts
  • correct choice of Boolean operators
  • accurate line numbers and absence of spelling errors
  • an appropriate text word search
  • inclusion of relevant subject headings
  • correct use of limits and filters
  • search strategy adaptations.

The importance of peer-review of search strings has been reported in three recent publications. In a study conducted by the Agency for Healthcare Research and Quality (AHRQ), which assessed the use of the PRESS checklist, the authors found that it “seems to cut down the time needed to do the review, increase response, and do a better job of identifying actual errors in search strategies” (5). In an internal investigation conducted by the Canadian Agency for Drugs and Technologies in Health (CADTH), the authors found that both the number and quality of the relevant articles retrieved in CADTH Rapid Responses reports were improved when using peer-review of search strings compared with no peer-review (6, 7).

At Source Health Economics (Source), to ensure our developed search strings for health technology assessment (HTA) submissions are comprehensive and of the highest quality, we incorporate external peer-review of search strings into all of the HTA SLRs we complete. As a result of Source’s strategic partnership with the School of Health and Related Research (ScHARR), Sheffield University, peer-review of our developed search strings is conducted by an information specialist at ScHARR, using the PRESS checklist (4).

References
  1. Lefebvre C, Duffy S. Peer reviewing search strategies. HTAi vortal 2018. Available at: http://vortal.htai.org/index.php?q=book/export/html/918.
  2. Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006;59(10):1057-63.
  3. Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Res Synth Methods. 2018;9(3):408-16.
  4. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-6.
  5. Relevo R, Paynter R. Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.) AHRQ Publication No. 12-EHC068-EF. Rockville, MD: Agency for HealthcareResearch and Quality. Accessed via: www.effectivehealthcare.ahrq.gov/reports/final.cfm. 2012.
  6. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at Canadian Health Libraries Association (CHLA) Conference; 22-25 May 2013; Saskatoon, Saskatchewan: Canada. Available at: https://www.cadth.ca/media/is/CSpry_Poster.pdf.
  7. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at the 21st Cochrane Colloquium; 19-23 Sep 2013; Quebec: Canada. Available at: https://abstracts.cochrane.org/2013-qu%C3%A9bec-city/peer-review-literature-search-strategies-does-it-make-difference.

 

At Source Health Economics, we are committed to delivering the highest quality search strings for HTA submissions. We’re proud to announce our partnership with ScHARR, where an information specialist reviews our developed search strings using the PRESS checklist.


More Insights

World Cancer Day 2025

World Cancer Day 2025

Today is World Cancer Day! The current campaign is ‘United by Unique’, which highlights the importance of personalised care to cater to each patient’s unique needs. In honour of World Cancer Day, Source has taken a look back at our 45 ... Read more

Manuscript deep dive

Manuscript deep dive

At Source, we pride ourselves on our tailored, strategic, and comprehensive approach to supporting our clients with their publications. Please click through the carousel below for more information, and please get in touch to discover how Source ... Read more

Sustainability and the future of HTAs

Sustainability and the future of HTAs

Exploring sustainability, climate change, and how including environmental data in HTAs could help healthcare reach sustainability goals Read more

World AIDS day 2024

World AIDS day 2024

Written by Emma Lones, Senior Medical Writer   This year, the campaign is focused on how individuals can help to end HIV stigma and make the world a better place for people living with HIV. In honour of World AIDS Day, Source has ... Read more