The importance of peer-review for your search strategy

Jan 2, 2019

Written by Neil Webb, Head of Systematic Review

The search strategy is the foundation of a well-conducted systematic literature review (SLR) and is the fundamental element that can affect the overall quality of the results of the final review (1, 2). The aim of a search string is to achieve a balance between sensitivity (capturing all relevant articles) and precision (eliminating irrelevant hits); a lack of balance has implications on the quality of the evidence identified and the resources required to conduct the review. In a recent study, a random sample of 70 new Cochrane SLRs published in 2015 were evaluated for the design and reporting of search strategies using recommended checklists. A problem in the design of the search was found in 73% (95% CI, 60-84%) of the reviews, and 53% (95% CI, 38-69%) of those problems could limit both the sensitivity and precision of the search strategies (3).

In addition, search string errors can result in the complete omission of highly relevant studies. Sampson et al 2006 found 82.5% of 105 MEDLINE search strings examined for errors contained an error that could potentially lower recall of relevant studies (2).

In order to enhance the quality of database search strings, peer-review feedback can be utilised. Peer-review of the search strategy provides a subjective validation, which can reduce the risk of errors, reduce the risk of missing relevant studies, and improve the balance between sensitivity and precision. The development of the PRESS checklist, an evidence based guideline in the form of a checklist, provides clear guidance for peer-reviewers to follow and ensures information specialists can check search strategies in a more formal, structured way (4). During development of the PRESS checklist, the following seven elements of the search strategy were highlighted as important to check when conducting a peer-review (2):

  • accurate translation of the research question into search concepts
  • correct choice of Boolean operators
  • accurate line numbers and absence of spelling errors
  • an appropriate text word search
  • inclusion of relevant subject headings
  • correct use of limits and filters
  • search strategy adaptations.

The importance of peer-review of search strings has been reported in three recent publications. In a study conducted by the Agency for Healthcare Research and Quality (AHRQ), which assessed the use of the PRESS checklist, the authors found that it “seems to cut down the time needed to do the review, increase response, and do a better job of identifying actual errors in search strategies” (5). In an internal investigation conducted by the Canadian Agency for Drugs and Technologies in Health (CADTH), the authors found that both the number and quality of the relevant articles retrieved in CADTH Rapid Responses reports were improved when using peer-review of search strings compared with no peer-review (6, 7).

At Source Health Economics (Source), to ensure our developed search strings for health technology assessment (HTA) submissions are comprehensive and of the highest quality, we incorporate external peer-review of search strings into all of the HTA SLRs we complete. As a result of Source’s strategic partnership with the School of Health and Related Research (ScHARR), Sheffield University, peer-review of our developed search strings is conducted by an information specialist at ScHARR, using the PRESS checklist (4).

References
  1. Lefebvre C, Duffy S. Peer reviewing search strategies. HTAi vortal 2018. Available at: http://vortal.htai.org/index.php?q=book/export/html/918.
  2. Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006;59(10):1057-63.
  3. Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Res Synth Methods. 2018;9(3):408-16.
  4. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-6.
  5. Relevo R, Paynter R. Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.) AHRQ Publication No. 12-EHC068-EF. Rockville, MD: Agency for HealthcareResearch and Quality. Accessed via: www.effectivehealthcare.ahrq.gov/reports/final.cfm. 2012.
  6. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at Canadian Health Libraries Association (CHLA) Conference; 22-25 May 2013; Saskatoon, Saskatchewan: Canada. Available at: https://www.cadth.ca/media/is/CSpry_Poster.pdf.
  7. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at the 21st Cochrane Colloquium; 19-23 Sep 2013; Quebec: Canada. Available at: https://abstracts.cochrane.org/2013-qu%C3%A9bec-city/peer-review-literature-search-strategies-does-it-make-difference.

 

At Source Health Economics, we are committed to delivering the highest quality search strings for HTA submissions. We’re proud to announce our partnership with ScHARR, where an information specialist reviews our developed search strings using the PRESS checklist.


More Insights

Obtaining a NICE recommendation for your medical device: a view from the external assessors

Obtaining a NICE recommendation for your medical device: a view from the external assessors

Written by Mark Pennington, Consultant Health Economist A National Institute for Health and Care Excellence (NICE) recommendation can open the door to widespread adoption for a new medical device. But does getting a NICE recommendation require ... Read more

10 steps for a successful global economic model

10 steps for a successful global economic model

Written by Beth Hancock, Senior Consultant Health Economist Pharmaceutical companies often develop global cost-effectiveness and budget impact models, with the aim of adapting them for specific countries. In principle, this is an efficient and ... Read more

Conceptual modelling and NICE requirements: a significant change or hill of beans?

Conceptual modelling and NICE requirements: a significant change or hill of beans?

Written by Alec Miners, Head of Health Economics The National Institute for Health and Care Excellence’s (NICE’s) new processes and methods guide includes several important changes to the way submitted economic evaluations should be ... Read more

British Heart Week 2022

British Heart Week 2022

As we approach the end of British Heart Week, we reflect on some of the key stats around heart conditions in the UK and our contribution as a company to help bring cutting-edge treatments to patients with cardiovascular diseases Read more