Majority of AI clinical trials report positive outcomes, yet concerns over generalizability persist

In a recent study published in The Lancet Digital Health, researchers examined the state of randomized controlled trials (RCTs) for artificial intelligence (AI) algorithms in clinical practice.

Study: Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review. Image Credit: metamorworks/Shutterstock.comStudy: Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review. Image Credit: metamorworks/Shutterstock.com

Background

The use of AI in healthcare has remarkably surged in the last five years, with some studies indicating that AI models could perform on par with or even better than clinicians. Many models have been evaluated retrospectively and not in real-world settings.

Of around 300 medical devices enabled with AI, some have been assessed in prospective RCTs. This scarcity contributes to uncertainty regarding the possibility of risk to clinicians and patients. Further, AI systems can perform poorly when prospectively deployed.

About the study

In the present study, researchers analyzed the current state of AI in clinical practice. They searched for relevant studies on the International Clinical Trials Registry and PubMed, CENTRAL, and SCOPUS databases between January 1, 2018, and November 14, 2023. References from studies were also screened to identify additional articles.

RCTs that implemented a substantial AI component as an intervention in clinical practice were eligible for inclusion. The intervention included non-linear computational models, i.e., neural networks, decision trees, etc.

Secondary studies, studies evaluating linear risk scores (logistic regression), and those not integrating the intervention into clinical practice were excluded. Abstracts/titles were screened, and full texts were reviewed.

Relevant data from eligible studies were extracted. These included participant characteristics, primary endpoint, clinical task(s), time efficiency endpoint, study location, comparator, AI type/origin, and results.

Studies were stratified by the primary endpoint group, clinical specialty, and AI data modality. Meta-analyses were not performed due to the heterogeneity in endpoints and tasks. Instead, an overview of trial features was presented.

Findings

The researchers identified 6,219 studies and 4,299 trial registrations. Following title/abstract screening, full texts of 133 studies were reviewed, which excluded 60 articles.

Reference screening identified 13 studies. Overall, 86 unique RCTs were included; 43%, 13%, 6%, and 5% of trials were related to gastroenterology, radiology, surgery, and cardiology, respectively.

Gastroenterology RCTs were notable for uniformity, as all trials tested video-based algorithms assisting clinicians. Further, only four groups (Fujifilm, Medtronic, Wuhan University, and Wision AI) conducted most (65%) gastroenterology trials.

In addition, 92% of RCTs were single-country trials undertaken primarily in the United States or China; conversely, six of the seven multi-country trials were conducted in European countries.

The median participant age was 57.3; 48.9% of subjects were male. Twenty-two RCTs reported race/ethnicity; the median proportion of White participants was 70.5%.

The primary endpoints in 46 trials were related to diagnostic performance or yield, such as mean absolute error and detection rate. Eighteen trials examined the effects of AI on care management. Fifteen AI algorithms evaluated patient symptoms and behavior.

Seven RCTs examined AI in clinical decision-making. Fifty-nine trials assessed deep learning models for medical imaging, predominately video-based rather than image-based. Others relied on structured data, i.e., health records, free text, and waveform data.

Most imaging-related AI systems were implemented in an assistive setup, whereas those based on structured data were compared with routine care.

Most models (55%) were developed in industry, followed by academia (41%). Eighty-one trials aimed to show improvement, 80% of which reported significant improvements in their primary endpoint.

Specifically, 46 trials observed improvements for clinicians assisted by AI systems compared to unassisted clinicians. Notably, three RCTs found that standalone AI systems performed better than clinicians. Five trials implemented non-inferiority designs.

Two trials examined non-inferiority between assisted and unassisted clinicians, and three assessed it between clinicians and standalone AI systems.

Overall, 70 trials reported favorable results for their primary endpoint. Sixteen RCTs had negative results, i.e., they found no improvements of assisted clinicians relative to unassisted clinicians, AI systems compared to routine care, and standalone AI models over clinicians.

Conclusions

Taken together, the findings reveal a growing interest in the utility of AI across clinical specialties and regions.

Most trials had favorable outcomes, underscoring the potential of AI systems in improving clinical decision-making, patient symptoms and behavior, and care management.

Notably, the success of AI ultimately depends on its generalizability to target populations and settings. Continued research is essential to deepen the understanding of AI's true effects and limitations.

Journal reference:
Tarun Sai Lomte

Written by

Tarun Sai Lomte

Tarun is a writer based in Hyderabad, India. He has a Master’s degree in Biotechnology from the University of Hyderabad and is enthusiastic about scientific research. He enjoys reading research papers and literature reviews and is passionate about writing.

Citations

Please use one of the following formats to cite this article in your essay, paper or report:

  • APA

    Sai Lomte, Tarun. (2024, April 29). Majority of AI clinical trials report positive outcomes, yet concerns over generalizability persist. News-Medical. Retrieved on November 04, 2024 from https://www.news-medical.net/news/20240429/Majority-of-AI-clinical-trials-report-positive-outcomes-yet-concerns-over-generalizability-persist.aspx.

  • MLA

    Sai Lomte, Tarun. "Majority of AI clinical trials report positive outcomes, yet concerns over generalizability persist". News-Medical. 04 November 2024. <https://www.news-medical.net/news/20240429/Majority-of-AI-clinical-trials-report-positive-outcomes-yet-concerns-over-generalizability-persist.aspx>.

  • Chicago

    Sai Lomte, Tarun. "Majority of AI clinical trials report positive outcomes, yet concerns over generalizability persist". News-Medical. https://www.news-medical.net/news/20240429/Majority-of-AI-clinical-trials-report-positive-outcomes-yet-concerns-over-generalizability-persist.aspx. (accessed November 04, 2024).

  • Harvard

    Sai Lomte, Tarun. 2024. Majority of AI clinical trials report positive outcomes, yet concerns over generalizability persist. News-Medical, viewed 04 November 2024, https://www.news-medical.net/news/20240429/Majority-of-AI-clinical-trials-report-positive-outcomes-yet-concerns-over-generalizability-persist.aspx.

Comments

The opinions expressed here are the views of the writer and do not necessarily reflect the views and opinions of News Medical.
Post a new comment
Post

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions.

You might also like...
Assessing and addressing IP staffing needs in a pediatric setting