The importance of peer-review for your search strategy

Written by Neil Webb, Head of Systematic Review

The search strategy is the foundation of a well-conducted systematic literature review (SLR) and is the fundamental element that can affect the overall quality of the results of the final review (1, 2). The aim of a search string is to achieve a balance between sensitivity (capturing all relevant articles) and precision (eliminating irrelevant hits); a lack of balance has implications on the quality of the evidence identified and the resources required to conduct the review. In a recent study, a random sample of 70 new Cochrane SLRs published in 2015 were evaluated for the design and reporting of search strategies using recommended checklists. A problem in the design of the search was found in 73% (95% CI, 60-84%) of the reviews, and 53% (95% CI, 38-69%) of those problems could limit both the sensitivity and precision of the search strategies (3).

In addition, search string errors can result in the complete omission of highly relevant studies. Sampson et al 2006 found 82.5% of 105 MEDLINE search strings examined for errors contained an error that could potentially lower recall of relevant studies (2).

In order to enhance the quality of database search strings, peer-review feedback can be utilised. Peer-review of the search strategy provides a subjective validation, which can reduce the risk of errors, reduce the risk of missing relevant studies, and improve the balance between sensitivity and precision. The development of the PRESS checklist, an evidence based guideline in the form of a checklist, provides clear guidance for peer-reviewers to follow and ensures information specialists can check search strategies in a more formal, structured way (4). During development of the PRESS checklist, the following seven elements of the search strategy were highlighted as important to check when conducting a peer-review (2):

• accurate translation of the research question into search concepts
• correct choice of Boolean operators
• accurate line numbers and absence of spelling errors
• an appropriate text word search
• inclusion of relevant subject headings
• correct use of limits and filters
• search strategy adaptations.

The importance of peer-review of search strings has been reported in three recent publications. In a study conducted by the Agency for Healthcare Research and Quality (AHRQ), which assessed the use of the PRESS checklist, the authors found that it “seems to cut down the time needed to do the review, increase response, and do a better job of identifying actual errors in search strategies” (5). In an internal investigation conducted by the Canadian Agency for Drugs and Technologies in Health (CADTH), the authors found that both the number and quality of the relevant articles retrieved in CADTH Rapid Responses reports were improved when using peer-review of search strings compared with no peer-review (6, 7).

At Source Health Economics (Source), to ensure our developed search strings for health technology assessment (HTA) submissions are comprehensive and of the highest quality, we incorporate external peer-review of search strings into all of the HTA SLRs we complete. As a result of Source’s strategic partnership with the School of Health and Related Research (ScHARR), Sheffield University, peer-review of our developed search strings is conducted by an information specialist at ScHARR, using the PRESS checklist (4).

References

1. Lefebvre C, Duffy S. Peer reviewing search strategies. HTAi vortal 2018. Available at: http://vortal.htai.org/index.php?q=book/export/html/918.
2. Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006;59(10):1057-63.
3. Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Res Synth Methods. 2018;9(3):408-16.
4. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-6.
5. Relevo R, Paynter R. Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.) AHRQ Publication No. 12-EHC068-EF. Rockville, MD: Agency for HealthcareResearch and Quality. Accessed via: www.effectivehealthcare.ahrq.gov/reports/final.cfm. 2012.
6. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at Canadian Health Libraries Association (CHLA) Conference; 22-25 May 2013; Saskatoon, Saskatchewan: Canada. Available at: https://www.cadth.ca/media/is/CSpry_Poster.pdf.
7. Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at the 21st Cochrane Colloquium; 19-23 Sep 2013; Quebec: Canada. Available at: https://abstracts.cochrane.org/2013-qu%C3%A9bec-city/peer-review-literature-search-strategies-does-it-make-difference.

At Source Health Economics, we are committed to delivering the highest quality search strings for HTA submissions. We’re proud to announce our partnership with ScHARR, where an information specialist reviews our developed search strings using the PRESS checklist.