Limiting conference abstract searching for systematic reviews

Written by Isobel Munro, Associate Systematic Review Analyst

The purpose of conducting a systematic literature review (SLR) is to identify a complete and unbiased set of studies relevant to the proposed research question. However, it is not only studies that have been fully published that are relevant to SLRs; conference abstracts contain a wealth of information that can also be incorporated if relevant. There are differing opinions on whether conference abstracts should be included in SLRs; in the National Institute for Health and Care Excellence (NICE) updated 2022 manual, NICE has made searching for conference abstracts optional when reviewing evidence for their guidelines and only recommend including them where there is good rationale to do so (1). Comparatively, the Cochrane handbook states that it is ‘highly desirable’ to search the relevant databases of conference abstracts (2), though it is left to the reviewer’s discretion to determine the methods of, and limits placed on, these searches.

Limiting an Embase search

It is common practice to limit Embase searches to identify conference abstracts published in the 3 years prior to the search date, whilst placing no limits on identification of full text articles. The assumption behind this is that 3 years is sufficient time for any data presented in conference abstracts to be published in full. However, this rationale is flawed; fewer than half of all studies first presented as conference abstracts are subsequently published in full, and a proportion of those take longer than 3 years to reach full publication (3,4). A Cochrane review conducted by Scherer et al, 2008, determined that the estimated publication rate of conference abstracts after 9 years was 52.6% for all studies, 63.1% for randomised controlled trials (RCTs), and 49.3% for other types of study design (5). Therefore, limiting your Embase search to a specific number of years (or placing no limits) will have consequences for your review.

Advantages of limiting an Embase search:

• Decreases resource requirements
• Conference abstracts do not always present consistent and dependable data. Most conference abstracts are not peer-reviewed and often only contain preliminary results. It is also common for there to be inconsistencies between data in abstracts and the article body of full texts published in journals (6).
• Conference abstracts do not always contain adequate information. There is frequently information missing from conference abstracts pertaining to the design, methods, outcomes, results, and risk of bias of a study. This also affects a study’s suitability to be included in any meta-analyses that may be subsequently undertaken.

 Disadvantages of limiting an Embase search:

• Introduces publication bias. Publication bias is when the likelihood of, or time to, publication of a study is impacted by the direction of the study’s results. Conference abstracts presenting ‘positive’ (statistically significant) results are more likely to be published than those with negative or neutral results (7).
• Introduction of other bias. There is an increased probability of a conference abstract being published in full if the abstract originates from an English-speaking country, and if it mentions randomisation (3,4).
• Limits often do not include a sufficient timespan. Trials with positive results were found to be published on average 4–5 years after they were first presented, whereas trials with null/negative results were found to be published on average 6–8 years after (2). Therefore, if conference abstract searches are date-limited, these findings should be taken into account.

Formal hand-searching for conference abstracts

When determining the appropriate methods for conference abstract searching, another consideration is whether it is necessary to formally hand-search for conference abstracts in addition to their identification using search strings via Embase. As of January 2022, 3.6 million conference abstracts from over 11,500 conferences have been indexed in Embase (8). Currently, hand-searching of any kind is not a requirement for Cochrane reviews and has the disadvantage of being costly and resource intensive (2). However, not all conferences are indexed in Embase. A small number of relevant conferences should therefore be hand-searched, although the same concerns regarding date limits also apply to formal hand-searching. Moreover, not all conference abstracts will have been indexed in a way that can be identified by the search strings used (2). Stovold et al, 2011, conducted a study in which they compared the relevant abstracts that were identified by hand-searching to those identified by Embase (9); the results showed that 7 out of the 34 (21%) relevant abstracts were not identified by the Embase search, highlighting the need to include formal hand-searching as part of the conference abstract identification process.

Suggested approaches

Some approaches for limiting conference abstract searches and their impact on the resulting SLR, and our clients are displayed in the table below:

†It is unfeasible to formally hand-search every conference abstract database, therefore hand-searching generally prioritises a limited number of databases based on the scope of the review.

Abbreviation: HTA, health technology assessment.

Conclusions

A reviewer can never be entirely certain that a sample of included studies is complete and fully representative of all the data collected for a specific set of criteria, especially when allowing for the constraints on time and budget that are generally present. However, by hand-searching databases for conference abstracts, and by not placing date limits on when these conference abstracts are published, the likelihood of conducting a complete SLR with a minimal risk of publication bias increases. An approach that strikes a balance between these two concepts is desirable.

If you would like to learn more about systematic literature reviews, please contact Source Health Economics, an independent consultancy specialising in evidence generation, health economics, and communication.

References

1. National Institute for Health and Care Excellence (NICE). 2022;
2. Higgins JP TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editor(s). Available from trainingcochraneorg/handbook.
3. Scherer RW et al. Cochrane Database Syst Rev. 2018;11:MR000005.
4. Diezel K PF, Adams CE. Psychological Medicine. 1999;29(2):491-4.
5. Scherer RW LP, von Elm E. The Cochrane Library. 2008;(4)
6. Pitkin RM et al. JAMA. 1999;281(12):1110-1.
7. Salami K et al. Pediatric Hematology and Oncology. 2013;30(3):165-9.
8. Embase content coverage. Accessed 07/01/2022, 2022. https://www.elsevier.com/solutions/embase-biomedical-research/coverage-and-content
9. Stovold E HS. Abstracts of the 19th Cochrane Colloquium; 2011 19-22 Oct; Madrid, Spain John Wiley & Sons; 2011.