Written by Hannah Shapiro
Quality assessment in single-arm trials
Quality assessment is a vital aspect of conducting a thorough systematic literature review (SLR), as the validity of the conclusions of the review depends on the reliability of the included literature. A poorly conducted study with bias in the methodology may introduce errors or skewed results. Therefore, quality assessment is important in minimising the risk of bias from included studies and to have confidence in the conclusions of the review. There are several quality assessment tools that are industry-standard for the appraisal of various study designs, such as randomised controlled trials (RCTs), observational studies, and SLRs. However, there are currently no quality assessment tools developed specifically for single-arm trials (SATs).
SATs are interventional trials with no comparative control group. While these trials are valuable for generating preliminary evidence, they are typically followed by more rigorous study designs, such as RCTs, to provide stronger evidence on the efficacy and safety of the intervention. However, in rare disease areas or new interventions, stronger comparative evidence may not yet be available and SATs may be the main source of evidence. In this case, it is important to appraise the quality of the evidence before using it in further analysis and decision-making.
When considering the most appropriate way to assess the quality of SATs, it is important to consider the potential sources of bias specific to the SAT design and the factors that contribute to the robustness of evidence.
The potential sources of bias specific to SATs include:
- Lack of control group: This makes it challenging to attribute observed outcomes solely to the intervention, so the generalisability of findings may be limited.
- Use of historical controls: Bias may arise due to changes in patient populations, treatment standards, or other external factors over time.
- Selection bias: Systemic bias in the selection of participants can affect the generalisability of the results to the broader population.
A quality assessment tool seeks to evaluate the methodological quality to mitigate these factors, as well as other common sources of bias that arise in all study types.
What are the current options?
The current options available for reviewers to assess SATs are imperfect. The first option is to not conduct a quality assessment, which can be acceptable in cases where there are RCTs and other more rigorous studies available as the main body of evidence. However, this is increasingly becoming a less appropriate option for health technology assessment (HTA) submissions as HTA agencies are more commonly requiring quality assessment for all included publications. Additionally, in many rare disease areas or in new interventions, evidence is limited and more rigorous studies may not be available for inclusion. In these instances, pre-existing quality assessment instruments may be used but may not be entirely suitable. Using a tool not specific to SAT designs will mean excluding sections regarding comparator groups and does not fully consider the potential biases associated with SAT designs.
Here we present some of the current quality assessment tools available for reviewers.
Table 1: Common risk of bias and quality assessment tools
| Quality assessment tool | Intended study designs | Tool description |
| Downs and Black Checklist | Randomised and non-randomised studies | Assesses risk of bias in 5 sections (reporting, external validity, bias, confounding, and study power). Recommended by Cochrane in previous editions of the manual. |
| ROBINS-I | Non-randomised comparative studies | Evaluates risk of bias in estimates of the comparative effectiveness of interventions from studies that did not randomise patients to arms. Recommended by Cochrane for non-randomised studies. |
| MINORS | Non-randomised surgical studies | 12 items designed to assess methodological quality of surgical studies but can be applied to other study types. The first 8 items are specific to non-comparative studies. |
| Newcastle-Ottawa Scale | Non-randomised studies, particularly observational and cohort studies | Assesses quality of selection, comparability, exposure, and outcome of study participants. Recommended by Cochrane in previous editions of the manual. |
| JBI Critical Appraisal Tools | Set of checklists for various study designs – e.g. cross-sectional, case-control, case series, RCTs | Focuses on assessment of internal validity and statistical conclusion validity. Each checklist is tailored to a different study design. |
Abbreviations: JBI, Joanna Briggs Institute; MINORS, Methodological index for non-randomized studies; RCT, randomised controlled trial; ROBINS-I, Risk Of Bias In Non-randomised Studies – of Interventions.
While these tools provide reasonable structured approaches for quality assessment, it is important to recognise their limitations when being adapted to SATs. These limitations include:
- Non-applicability of key domains relating to comparator groups
- Comparative nature of the tool
- Inadequate consideration of confounding
- Scoring system limitations when leaving out questions
- Limited guidance on historical controls
- Lack of specific criteria related to SAT design
- Limited focus on external validity.
Our recommendation
Recently, we have seen an increased requirement for the quality assessment of SATs from HTA agencies in submissions, regardless of the amount of additional included publications. Therefore, we would recommend using a pre-existing quality assessment tool to conduct a quality assessment of any SATs included in a review. Particularly, we have found the Downs and Black checklist most adaptable to the SAT design. However, we are seeing an increasing demand for an industry-standard quality assessment tool specifically designed to consider the unique characteristics of SATs. This is something that the team at Source Health Economics are currently working towards developing.
In conclusion, whether using a pre-existing tool or a novel design-specific tool, it is most important to be transparent with the limitations in the applicability of the tool to SATs and to clearly justify any adaptations in the methodology of the systematic review.
If you would like to learn more about SLRs, please contact Source Health Economics, an independent consultancy specialising in evidence generation, health economics, and communication.
