Quality assessment in single-arm trials: insights for systematic reviews

Apr 9, 2024

Written by Hannah Shapiro


Quality assessment in single-arm trials

Quality assessment is a vital aspect of conducting a thorough systematic literature review (SLR), as the validity of the conclusions of the review depends on the reliability of the included literature. A poorly conducted study with bias in the methodology may introduce errors or skewed results. Therefore, quality assessment is important in minimising the risk of bias from included studies and to have confidence in the conclusions of the review. There are several quality assessment tools that are industry-standard for the appraisal of various study designs, such as randomised controlled trials (RCTs), observational studies, and SLRs. However, there are currently no quality assessment tools developed specifically for single-arm trials (SATs).

SATs are interventional trials with no comparative control group. While these trials are valuable for generating preliminary evidence, they are typically followed by more rigorous study designs, such as RCTs, to provide stronger evidence on the efficacy and safety of the intervention. However, in rare disease areas or new interventions, stronger comparative evidence may not yet be available and SATs may be the main source of evidence. In this case, it is important to appraise the quality of the evidence before using it in further analysis and decision-making.

When considering the most appropriate way to assess the quality of SATs, it is important to consider the potential sources of bias specific to the SAT design and the factors that contribute to the robustness of evidence.

The potential sources of bias specific to SATs include:

  • Lack of control group: This makes it challenging to attribute observed outcomes solely to the intervention, so the generalisability of findings may be limited.
  • Use of historical controls: Bias may arise due to changes in patient populations, treatment standards, or other external factors over time.
  • Selection bias: Systemic bias in the selection of participants can affect the generalisability of the results to the broader population.

A quality assessment tool seeks to evaluate the methodological quality to mitigate these factors, as well as other common sources of bias that arise in all study types.


What are the current options?

The current options available for reviewers to assess SATs are imperfect. The first option is to not conduct a quality assessment, which can be acceptable in cases where there are RCTs and other more rigorous studies available as the main body of evidence. However, this is increasingly becoming a less appropriate option for health technology assessment (HTA) submissions as HTA agencies are more commonly requiring quality assessment for all included publications. Additionally, in many rare disease areas or in new interventions, evidence is limited and more rigorous studies may not be available for inclusion. In these instances, pre-existing quality assessment instruments may be used but may not be entirely suitable. Using a tool not specific to SAT designs will mean excluding sections regarding comparator groups and does not fully consider the potential biases associated with SAT designs.

Here we present some of the current quality assessment tools available for reviewers.



Table 1: Common risk of bias and quality assessment tools

Quality assessment tool Intended study designs Tool description
Downs and Black Checklist Randomised and non-randomised studies Assesses risk of bias in 5 sections (reporting, external validity, bias, confounding, and study power). Recommended by Cochrane in previous editions of the manual.
ROBINS-I Non-randomised comparative studies Evaluates risk of bias in estimates of the comparative effectiveness of interventions from studies that did not randomise patients to arms. Recommended by Cochrane for non-randomised studies.
MINORS Non-randomised surgical studies 12 items designed to assess methodological quality of surgical studies but can be applied to other study types. The first 8 items are specific to non-comparative studies.
Newcastle-Ottawa Scale Non-randomised studies, particularly observational and cohort studies Assesses quality of selection, comparability, exposure, and outcome of study participants. Recommended by Cochrane in previous editions of the manual.
JBI Critical Appraisal Tools Set of checklists for various study designs – e.g. cross-sectional, case-control, case series, RCTs Focuses on assessment of internal validity and statistical conclusion validity. Each checklist is tailored to a different study design.

Abbreviations: JBI, Joanna Briggs Institute; MINORS, Methodological index for non-randomized studies; RCT, randomised controlled trial; ROBINS-I, Risk Of Bias In Non-randomised Studies – of Interventions.


While these tools provide reasonable structured approaches for quality assessment, it is important to recognise their limitations when being adapted to SATs. These limitations include:

  • Non-applicability of key domains relating to comparator groups
  • Comparative nature of the tool
  • Inadequate consideration of confounding
  • Scoring system limitations when leaving out questions
  • Limited guidance on historical controls
  • Lack of specific criteria related to SAT design
  • Limited focus on external validity.


Our recommendation

Recently, we have seen an increased requirement for the quality assessment of SATs from HTA agencies in submissions, regardless of the amount of additional included publications. Therefore, we would recommend using a pre-existing quality assessment tool to conduct a quality assessment of any SATs included in a review. Particularly, we have found the Downs and Black checklist most adaptable to the SAT design. However, we are seeing an increasing demand for an industry-standard quality assessment tool specifically designed to consider the unique characteristics of SATs. This is something that the team at Source Health Economics are currently working towards developing.

In conclusion, whether using a pre-existing tool or a novel design-specific tool, it is most important to be transparent with the limitations in the applicability of the tool to SATs and to clearly justify any adaptations in the methodology of the systematic review.

If you would like to learn more about SLRs, please contact Source Health Economics, an independent consultancy specialising in evidence generation, health economics, and communication.

More Insights

A ‘NICE’ Innovation: The Proportionate Approach

A ‘NICE’ Innovation: The Proportionate Approach

Written by Kiera Lander, Assistant Project Manager   The National Institute for Health and Care Excellence (NICE) has long been at the forefront of health technology assessment (HTA), ensuring that patients have access to innovative, ... Read more

Out of the frying pan and into the fire: Moving to a market access consultancy company

Out of the frying pan and into the fire: Moving to a market access consultancy company

Written by Brian O'Toole, HTA Writing Lead   Last year I changed jobs. It was a major life decision that took time to make as it meant leaving the comfort and familiarity of academia for the hustle and bustle of a market access ... Read more

AI-enabled triage in the NHS: an economic analysis

AI-enabled triage in the NHS: an economic analysis

Written by Mark Pennington, Consultant Health Economist   With the UK’s National Health Service (NHS) facing an unprecedented patient demand for healthcare, there is increasing interest in artificial intelligence (AI) and how it can ... Read more

Modelling requirements for HTA in Europe’s big 5 economies

Modelling requirements for HTA in Europe’s big 5 economies

Written by Oliver Burn, Consultant Health Economist, and Margaret Petrou, Consultant Health Economist Adapting a global economic model for Health Technology Assessment (HTA) submissions in several countries can be challenging, as requirements ... Read more