AI models show promise in predicting heart disease risks, but lack validation

By Dr. Priyom Bose, Ph.D.Feb 8 2024Reviewed by Benedette Cuffari, M.Sc.

In a recent review published in BMC Medicine, scientists evaluate artificial intelligence models (AI-Ms) that predict cardiovascular disease (CVD) risks in general and specific populations while also developing an independent validation score (IVS) for AI-Ms.

Study: Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review. Image Credit: Summit Art Creations / Shutterstock.com

Background

The global prevalence of cardiovascular diseases (CVDs) is increasing rapidly, which has led to the development of several CVD prediction models. CVD prediction models like Framingham and SCORE identify individuals at a greater risk of developing CVDs to ultimately implement preventive measures across the vulnerable population.

Within computer science, the application of AI, machine learning (ML), and deep learning (DL) can be used to develop computational systems with a similar functioning capacity analogous to human intelligence while performing a complex task. This functioning capacity is associated with humans' reasoning, learning, perception, problem-solving, decision-making, and language comprehension skills.

AI-Ms have been increasingly applied in the healthcare sector for disease risk prediction. However, this application has been subjected to multiple challenges linked to data privacy, security, transparency, legality, and concerns related to ethics. Nevertheless, as compared to traditional risk prediction models, AI-Ms are associated with greater accuracy, data-processing capability, and fewer processing restrictions. 

About the study

Extensive data extraction was performed based on predictors, algorithms, bias, and population. A tool to assess the replicability and applicability of AI-Ms, as well as ensure the external validation of AI-Ms, was developed to screen AI-Ms.

For the current review, all relevant articles were obtained from Embase, Web of Science, PubMed, and IEEE Library. The prediction risk of bias assessment tool (PROBAST) was also used. 

Key findings

A total of 79 relevant articles published between 2017 and 2021 were obtained, of which 486 AI-Ms were identified. Most of these studies were related to the development of new AI-Ms; however, none of the models underwent independent external validation.

Thus, AI risk prediction researchers appear to be more focused on developing new models than validating existing ones, which is crucial for clinical applications. Since unvalidated AI-Ms would result in the generation of many useless prediction models, researchers must focus on validating AI-Ms to avoid wasting research time.

A key factor that restricts the implementation of external validation is the use of limited data sources for model development. However, this could be addressed by using data from multi-source databases. 

Most AI-based models as CVD risk predictors were developed in North America and Europe, very few of which were developed in Asian and South American countries, whereas none were developed in Africa. Since the extent of CVD risks varies among ethnicities, it is important to develop AI-Ms that focus on specific ethnic groups.

The four most common variables used in AI-Ms for CVD risk predictions include total cholesterol, age, sex, and smoking status. Compared to traditional models, AI-Ms evaluate multimodal data, including additional gene- or protein-related information and image data. Other advantages of AI models include data re-input and utility.

Many studies did not provide important research information, which compromised model validation. In the future, studies must provide a Transparent Reporting of a multivariable prediction model for the Individual Prognosis Or Diagnosis (TRIPOD) statement when the manuscript is submitted.

According to PROBAST, all models were at a high risk of bias, primarily because of the inappropriate use of statistical tools. IVS analysis revealed that only 10 models were "recommended" for use, whereas the remaining models were categorized under "not recommended" or "warning."

The IVS tool has been developed for screening independent external validation models. This scoring system evaluates the suitability for independent external validation based on transparency, risk assessment, performance, and clinical implication.

The newly developed IVS indicated that independent external validation may not be suitable for over 95% of the models, thus implying that these models cannot be used in clinical settings.

Conclusions

Although several AI-Ms for CVD predictions are available, few studies have systematically analyzed the models for their effectiveness. The current review summarized AI-Ms for CVD and discussed current challenges associated with their use.

The current study provided important insights into AI models used for CVD risk predictions, including the geographical imbalance, a high risk of bias, a low standard-reaching rate of report quality, a lack of independent external validation, and an imperfect evaluation system. In this context, the use of a newly developed IVS tool could help assess the replicability of the models.