Risk of bias in evidence underpinning approval of new cancer drugs raises questions
- 19 September 2019
Around half of trials that supported new cancer drug approvals in Europe between 2014 and 2016 were judged to be at high risk of bias, a study published by the British Medical Journal (BMJ) has found. The research was led by the London School of Economics, with methodological input from the National Institute for Health Research (NIHR) Bristol Biomedical Research Centre (BRC) and NIHR Collaboration for Leadership in Applied Health Research and Care West (CLAHRC West).
The European Medicines Agency (EMA) is responsible for evaluating the clinical effectiveness and safety of new medicines. In 2017, more than a quarter (24 of 92) of EMA approvals were for cancer drugs, most of which were based on evidence from randomised controlled trials, considered to be the “gold standard” for evaluating treatment effectiveness.
However, flaws in the design, conduct, analysis, or reporting of randomised controlled trials can distort estimates of treatment effect. To evaluate these flaws in more detail, the researchers examined the design, risk of bias, and reporting of randomised controlled trials that supported European approvals of cancer drugs from 2014 to 2016. The risk of bias was assessed using the new Risk of Bias tool for randomised trials (RoB 2) developed by researchers from NIHR CLAHRC West and NIHR Bristol Biomedical Research Centre. This tool assesses several domains of trial design, conduct, analysis and reporting; if any of these components is considered to be associated with a high risk of bias, the study result overall is also considered at high risk of bias.
During this period, the EMA approved 32 new cancer drugs on the basis of 54 studies. Of these, 41 (76%) studies were randomised controlled trials; 39 had available publications and were therefore included in the study.
Only 10 trials (26%) measured overall survival as a main (primary) endpoint. The remaining 29 trials (74%) evaluated indirect (surrogate) measures of clinical benefit, which do not always reliably predict whether a patient will live longer or have a better quality of life.
Overall, 19 trials (49%) were judged to be at high risk of bias because of deficits in their design, conduct, or analysis. Trials that evaluated overall survival were at lower risk of bias than those that evaluated surrogate measures of clinical benefit.
The researchers point out that their review of the evidence has some limitations. For example, they did not include clinical study reports, which contain detailed information about trial methods and results, and they focused only on cancer drug trials, so findings may not apply to trials in other therapeutic areas. In addition, they evaluated “risk” of bias rather than bias itself: it remains a possibility that the methodological deficits identified by the authors did not lead to biased findings.
They also acknowledge that some risk of bias might be unavoidable because of the complexity of cancer trials. For example, in more than half (10 out of 19) of the trials rated high risk of bias, the reason for the high risk rating was the missing outcome data, which often cannot be avoided even in well-designed trials. The authors say that their findings should prompt policymakers, investigators, and clinicians “to carefully consider risk of bias in pivotal trials that support regulatory decisions, and the extent to which new cancer therapies offer meaningful benefit to patients”.
Julian Higgins, Professor of Evidence Synthesis at the University of Bristol Medical School, said:
“Our findings show that these studies are at risk of bias, but this doesn’t necessarily mean that they are biased. It can be difficult to avoid some of the issues that we’ve identified, such as missing outcome data, in even the most well-designed studies. Policymakers and regulators should carefully consider risk of bias in trials that support regulatory decisions.”
Paper
Huseyin Naci, Courtney Davis, Jelena Savović, Julian P T Higgins, Jonathan A C Sterne, Bishal Gyawali, Xochitl Romo-Sandoval, Nicola Handley, Christopher M Booth
Published in the BMJ