Utilizing deep learning to enhance the accuracy of mortality forecasts in emergency rooms

By Susha Cheriyedath, M.Sc.Sep 14 2023Reviewed by Sophia Coveney

In a recent study published in Scientific Reports, a group of researchers enhanced the accuracy of mortality predictions in emergency departments (EDs) by employing advanced data-synthesis techniques and machine learning models.

There was a primary focus on improving the F1 score while retaining a high Area Under the Curve (AUC) score, using a dataset from Yonsei Severance Hospital's ED.

Background 

Every year, US EDs accommodate 130 million visits, resulting in resource strains and crowding, a situation worsened by the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) pandemic.

The current triage systems are subjective and prone to errors. Machine learning (ML) can improve accuracy in predicting patient outcomes, but early models had limitations. Further research is essential to optimize ML and data-synthesis algorithms for mortality predictions in EDs, addressing dataset imbalances and feature efficacy.

About the study 

The present study utilized data from 7,325 individuals who visited Yonsei Severance Hospital’s ED in Seoul, South Korea between January and June 2020. As a designated coronavirus disease 2019 (COVID-19) screening clinic, the hospital was tasked with managing severe cases. The data was collected by authorized medical personnel via the hospital’s electronic system. 

Twenty-one features were utilized for analysis. Six features (nebulizer, chest X-ray, O₂ apply, blood test, fluid, and medication) indicated whether patients received specific treatments post-initial evaluation.

High blood pressure status (HiBP),  diabetes mellitus (DM), allergy, pulmonary tuberculosis (Pul. Tbc), hepatitis, and other medications were based on the medical history of the patient.

The final set of seven metrics, including mental status and vital signs, was acquired during the initial evaluation. For example, the blood test metric indicated a complete blood count test, while HiBP revealed high blood pressure presence.

The initial dataset had 7,325 patients, but 1,543 records had missing features, leading to a refined dataset of 5,782 records. For the machine learning system’s training and evaluation, data was divided into training and test sets.

Given the dataset’s inherent imbalance, data-synthesis techniques like Synthetic Minority Over-sampling Technique (SMOTE) and Conditional Tabular Generative Adversarial Network (CTGAN) were used to generate synthetic deceased patient data.

For prediction, four machine learning prediction algorithms were employed, ranging from traditional machine learning to Deep Neural Network (DNN)-based learning. Despite DNN’s general underperformance for tabular datasets, TabNet was used for its recent superior performance.

The prediction models framework consisted of preprocessing, data division, augmentation through data-synthesis algorithms, and training multiple classification models. An ensemble approach was finally adopted to merge model predictions.

The present study emphasized the F1 score as the primary evaluation metric over conventional accuracy scores due to its relevance in imbalanced medical datasets.

Study results

The central objective of the study was to pinpoint the most effective blend of machine learning (ML) classification models and data synthesis techniques to accurately predict the mortality rates of patients in the ED. Given the imbalanced nature of the dataset, the F1 score was selected as the chief performance criterion.

The top five models displayed noteworthy performance metrics, such as the F1 score, AUC, accuracy, precision, and recall. Notably, the leading model, which employed the Gaussian Copula for data synthesis combined with the CatBoost classifier, stood out in its predictive prowess.

This model showcased an AUC of 0.9731, an F1 score of 0.7059, and an impressive accuracy rate of 0.9914. Moreover, its precision stood at 0.8000 while its recall was 0.6316. The commendable recall value is particularly significant as it implies the model can proficiently identify the positive class, representing the most urgent patients in dire need of medical intervention.

When assessing the broad scope of results, it is evident that different mixes of ML algorithms and data-synthesis methods yielded commendable results in predicting patient mortality in the ED. The top-tier models consistently achieved high scores across various performance metrics.

These outcomes underscore the tremendous promise ML models hold in enhancing predictions regarding patient outcomes in emergency healthcare settings. Such advancements can pave the way for healthcare practitioners to make well-informed decisions, ultimately leading to timely and fitting medical interventions.

This not only augments the efficiency of medical procedures but could potentially save lives, emphasizing the profound impact of integrating advanced predictive models into the healthcare domain.

Conclusions

To summarize, the present study introduced twenty-one distinct features that surpassed prior benchmarks in predicting mortality in emergency departments. Despite challenges with imbalanced datasets, the model achieved a notably high F1 score, indicating reliable predictive capabilities.

When compared to conventional triage systems and past research, this study’s models, especially those utilizing synthetic data from the Gaussian Copula method, showed superior performance.

The variance in traditional triage scores highlighted the need for consistent, intelligent systems in healthcare. The study’s data-synthesis algorithm effectively enhanced model predictions, underlining its importance in training machine learning models.