Language restrictions in systematic literature reviews: a reflection on the current tone

Written by Alys Ridsdale, Associate Systematic Review Analyst

 

Introduction

There are over 7,100 spoken languages across the globe (1). Despite this linguistic diversity, including language restrictions in a systematic literature review (SLR) is common practice (2). The main aim of an SLR is to identify all information relevant to the research question in an unbiased manner (3). As such, The Cochrane Collaboration recommend SLRs include non-English language studies with justifications required for limiting to English publications (4). However, search strategies and eligibility criteria often restrict SLRs to studies published in English (2). This raises concerns over limiting SLRs to English publications due to the potential of language bias, which may impact SLR findings (2, 4, 5). For this reason, the use of language limits may appear counter-intuitive, raising questions over their acceptability (2, 5).

 

How might we justify language restrictions?

Frequently reported barriers to including non-English language studies in SLRs include (5):

• • Insufficient funding
  • Lack of time
  • Difficulty locating non-English publications
  • Difficulty in article translation and interpretation by reviewers
  • Availability and cost of professional translators.

These logistical challenges of searching and assessing non-English language studies highlight some of the many practical and monetary considerations when developing your SLR strategy. Furthermore, issues in publication interpretation might be factors for concern (2). However, as English has become the central scientific language, forming the written voice of up to 98% of scientific publications (6, 7), including only English language studies within SLRs may enable reviewers to capture and summarise most of the available data in a timely and cost-efficient manner.

 

Implications of SLR language restrictions

There is conflicting evidence on whether language restrictions increase the risk of language bias within SLRs (8). Egger et al, 1997 found that German-speaking authors were more likely to publish in English if their results were statistically significant (9). Such alteration of publication language based on study findings (termed language bias) could result in SLRs overlooking non-significant, but important, studies (2, 4, 5). In contrast, Nussbaumer-Streit et al, 2020 showed that excluding non-English publications did not markedly change the conclusions of any of the 40 randomly selected Cochrane reviews (10). Hence, while language limits reduce outcome certainty, SLR conclusions may not always be affected (11). There is a balance to be found between the additional insights non-English studies may bring against the additional time and cost of their discovery and translation (12). This is also likely to be dependent on the indication and scope of the review (2). The decision to exclude or include non-English language studies should therefore be independently considered for each SLR.

Where language restrictions are justified, Cochrane and other collaborations place emphasis on limiting their extent where practical and ensuring clear reporting (2). For example, SLRs could set protocols including searches of international trial registers and databases that may hold English translations of publications (2).

 

Potential changes to translation toolkits

As translation technologies like Google translate and DeepL continue to improve, so does the ability to screen and assess publications of different languages and dialects (13). Use of artificial intelligence (AI) may also permit high-quality, affordable translation in the not-too-distant future (1). However, these technologies are not without error and their utility in scientific communication is unclear (2, 14). In a context where small translation errors could lead to considerable differences in the conclusions drawn, these tools may not present a sensible option to assess the full text of a publication (2). Nevertheless, they could prove useful to screen non-English data sources and offer a future avenue for efficient, low-cost translation (2).

 

Conclusion

For many SLRs, language restrictions represent a viable option to manage the contrasting priorities of resources and knowledge (2). The implementation of language restrictions with clear justification can minimise translation and data interpretation challenges, thus promoting them as an efficient and effective practice (6, 8).

Ever-advancing technology and improved global communication however may tip these scales. Continuous review on the use of language limits and clear reporting will increase the transparency and quality of future SLRs.

 

If you would like to learn more about systematic literature reviews, please contact Source Health Economics, a HEOR consultancy specialising in evidence generation, health economics, and communication.

 

References

1. Bahji A. Nordisk Alkohol Nark. 2023;40(1):6-13.
2. Stern C. JBI Evidence Synthesis. 2020;18(9):1818-9.
3. Thomas C. Available at: https://source-he.com/pros-and-cons-of-crowdsourcing-for-systematic-review/ (last accessed 01 Jul 2024). 2024.
4. Cochrane Collaboration. Available at: https://training.cochrane.org/handbook/current/chapter-04 (last accessed 31 Jul 2024). 2023.
5. Neimann Rasmussen L. Syst Rev. 2018;7(1):129.
6. Ramírez-Castañeda V. PLOS ONE. 2020;15(9):e0238372.
7. Steigerwald E. Bioscience. 2022;72(10):988-98.
8. Helbach J. BMC Medical Research Methodology. 2022;22(1):230.
9. Egger M. 1997;350(9074):326-9.
10. Nussbaumer-Streit B. J Clin Epidemiol. 2020;118:42-54.
11. de Kock S. Presented as part of Cochrane’s 2020 annual scientific meeting. Available at: https://abstracts.cochrane.org/2020-abstracts/potential-impact-english-language-limits-systematic-reviews. 2020.
12. Walpole SC. J Clin Epidemiol. 2019;111:127-34.
13. Mahmić-Kaknjo M. Syst Rev. 2023;12(1):56.
14. Aiken M. Studies in Linguistics and Literature. 2019;3:p253.