Improving your search strategy: randomised controlled trial filters (1/2)

Written by Daisy Stewart (Systematic Review Analyst) and Tom Macmillan (Consultant - Systematic Review)

A search filter is a pre-written search strategy (string of search terms) that is designed to retrieve studies with a particular methodology or focus from a specific database and platform (1). The most common filters are designed to retrieve randomised controlled trials (RCTs), which are widely considered the most robust form of research for evidence-based medicine (2, 3).

Filters can easily be applied to your own search strategy to increase its sensitivity (ensure more relevant records are retrieved) and precision (retrieve fewer irrelevant records) (4). In addition, filters improve the consistency of the search process and demonstrate a recognised standard (4).

Some databases feature built-in filters that allow search results to be limited by design and/or focus. For example, databases on the Ovid platform include publication type field limits, and PubMed Clinical Queries limits search results to specific clinical research areas (5). However, health technology assessment (HTA) bodies, including the National Institute for Health and Care Excellence (NICE), do not recommend this approach and advise using a validated RCT filter instead to ensure that the search strategy is transparent and reproducible (6).

The validation process

Filters are developed by selecting search terms and subject headings (manually or using statistical methods) and testing these against a “gold standard” reference set of known records, which is created by relative recall or hand-searching (7, 8).

High-quality filters should also be validated to evaluate their effectiveness by using a set of known relevant records that are independent from the gold standard used to derive and test the filter (7).

Filters are typically designed to maximise sensitivity or precision (3). Filters that achieve high sensitivity, the most frequently reported type of filter (9), typically do so at the cost of lower precision, and vice versa (4). Therefore, highly sensitive filters may retrieve a large number of records that need to be screened to identify those that are relevant, while highly precise filters may fail to capture all relevant records.

Optimum filters maximise sensitivity while maintaining a sufficient level of precision (10). Information specialists define high sensitivity as >95% sensitivity and 20% precision, and moderate sensitivity as ≥80% sensitivity and ≥20% precision (11). However, a level of precision below 20% may still be adequate since it is likely to be increased when the filter is combined with an effective search strategy (12).

How to use a search filter

Generally, a search filter is simply copied and pasted into the database search box and combined with the search strategy with the Boolean operator ‘AND’, like this (13):

1. Subject search terms
2. RCT filter
3. 1 AND 2

It is recommended to save your copied-and-pasted search filter as a separate search strategy, so that it can be easily added to future searches and shared with colleagues (14).

Beware that some databases are pre-filtered. For example, Cochrane’s CENTRAL is pre-filtered to contain only reports with study designs that are relevant for inclusion in Cochrane Reviews, therefore there is no need to apply an RCT filter when searching this database (15).

Where to find an RCT search filter

There are many validated RCT filters available for different databases and platforms. Cochrane recently funded the translation of their highly sensitive RCT filter for use in Ovid to Embase.com (16). In addition, Chapter 4 of the Cochrane Library Handbook lists RCT filters for the Medline database on both Ovid and PubMed (1).

Furthermore, the NICE InterTASC Information Specialists’ Sub-group (ISSG) Search Filter Resource offers a comprehensive collection of search filters, including those for RCTs, and provides critical appraisals of several of these (17).

How to choose a filter

When identifying RCTs to be referenced in HTAs, the recommendations of the HTA body they are intended for should be considered. Some HTA bodies do not explicitly recommend the use of RCT filters while those that do, including NICE and the Scottish Medicines Consortium (SMC), leave the choice of filter to the discretion of the searcher (6, 18). NICE reports that this is because there is often limited evidence on the performance of individual filters (19).

To make an informed choice of filter, it is important to consider its age (to take into account database indexing or interface changes); its relative levels of sensitivity, specificity and precision; and the reliability of its development and reported performance, including how recently and thoroughly it was validated or appraised (1, 17).

It may be helpful to refer to performance reviews of published filters, which contain critical appraisals of the focus, design, testing, limitations, and comparisons of filters. McKibbon et al. (2009) reviewed the performance of 38 RCT filters (3). In addition, the ISSG has published a search filter critical appraisal checklist as a comprehensive tool to assist in filter selection (20). Both sources may be useful when selecting a filter.

Nonetheless, validated filters should be used with caution. Study design and other concepts may not be explicitly stated within the title or abstract of a database record, and may not be captured by the indexing, meaning filters may fail to retrieve these records (7). In addition, it has been demonstrated that there is considerable variability in the extent to which filters are tested and validated (7). This may be due to a lack of standardised definition of ‘validation’ within search filter methodology, highlighting a need for more explicit methods for reliable and consistent validation, which will ultimately improve the speed and efficacy of literature searching.

If you would like to learn more, please contact the Evidence Generation team at Source Health Economics, an independent consultancy specialising in evidence generation, health economics, and communication.

 

Bibliography

1. Cochrane Handbook for Systematic Reviews of Interventions version 6.1. Chapter 4: Searching for and selecting studies: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors); [updated September 2020. Available from: www.training.cochrane.org/handbook.
2. Sibbald B, Roland M. Understanding controlled trials. Why are randomised controlled trials important? BMJ. 1998;316(7126):201.
3. McKibbon KA, Wilczynski N, Haynes RB. Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. Health Info Libr, J. 2009;26(3):187-202.
4. Beale S, Duffy S, Glanville J, Lefebvre C, Wright D, McCool R, et al. Choosing and using methodological search filters: searchers’ views. Health and Information Libraries. 2014;31(2):133-47.
5. PubMed Clinical Queries Ncbi.nlm.nih.gov: Maryland: National Library of Medicine; [Available from: https://www.ncbi.nlm.nih.gov/pubmed/clinical.
6. 5 Identifying The Evidence: Literature Searching And Evidence Submission. Developing NICE Guidelines: The Manual (PMG20). 2014:95.
7. Jenkins M. Evaluation of methodological search filters – a review. Health and Information Libraries. 2004;21(3):148-63.
8. Sampson M ZL, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, et al. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Med Res Methodol 2006;6(33).
9. Harbour J, Fraser C, Lefebvre C, Glanville J, Beale S, Boachie C, et al. Reporting methodological search filter performance comparisons: a literature review. Health Information and Libraries Journal. 2014;31(3):176-94.
10. Lunny C, Salzwedel DM, Liu T, Ramasubbu C, Gerrish S, Puil L, et al. Validation of five search filters for retrieval of clinical practice guidelines produced low precision. Journal of Clinical Epidemiology. 2020;117:109-16.
11. J G, K F, A Y, D K, S M. Development and Testing of Search Filters to Identify Economic Evaluations in MEDLINE and EMBASE. Ottawa: Canadian Agency for Drugs and Technologies in Health. 2009.
12. Glanville J FK, Yellowlees A, Kaunelis D, Mensinkai S. Development and Testing of Search Filters to Identify Economic Evaluations in MEDLINE and EMBASE. Ottawa: Canadian Agency for Drugs and Technologies in Health. 2009.
13. Guides: EPIB 619 Systematic Reviews & Meta-Analyses: Search filters for RCTs and more Libraryguides.mcgill.ca: McGill Library; [updated 25 January 2021. Available from: https://libraryguides.mcgill.ca/epib619/rct-filters.
14. Searching for Systematic Reviews: Using Filter Libguides.kcl.ac.uk: King’s College London; [updated 19 January 2021. Available from: https://libguides.kcl.ac.uk/systematicreview/SRFilters.
15. How CENTRAL is created. Cochrane Library [Available from: https://www.cochranelibrary.com/central/central-creation.
16. Glanville J, Foxlee R, Wisniewski S, Noel-Storr A, Edwards M, Dooley G. Translating the Cochrane EMBASE RCT filter from the Ovid interface to Embase.com: a case study. Health Information and Libraries Journal. 2019(36):264-77.
17. ISSG Search Filter Resource [Internet] York (UK): Glanville J, Lefebvre C, Manson P, Robinson S and Shaw N, editors; 2006 [updated 26 February 201819 January 2021]. Available from: https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home.
18. Guidance to manufacturers for completion of New Product Assessment Form (NPAF): Scottish Medicines Consortium; November 2017.
19. 5 Identifying The Evidence: Literature Searching And Evidence Submission. Developing NICE Guidelines: The Manual (PMG20). 2014:95-6.
20. Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman N, et al. So many filters, so little time: the development of a search filter appraisal checklist. J Med Libr Assoc. 2008;96(4):356-61.