RapiD_AI as a novel disease and pandemic preparedness tool

By Bhavana KunkalikarAug 16 2022Reviewed by Danielle Ellis, B.Sc.

In a recent study posted to the medRxiv* preprint server, researchers assessed the RapiD_AI framework for deployable artificial intelligence (AI) for improved pandemic preparedness.

*Important notice: medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Background

The coronavirus disease 2019 (COVID-19) pandemic will probably not be the last pandemic the human race will face. Research studying historical pandemics recorded from the 1600s to the present has revealed that the risk of the incidence of a COVID-19-like pandemic has a probability of 2.63% annually and 38% in a lifetime. While it is difficult to prevent such pandemics in the future, it is under our control to be prepared for their adverse impacts with appropriate preparedness. 

About the study

In the present study, researchers developed a framework called RapiD_AI that could guide the usage of pre-trained neural network models as a tool for improving pandemic preparedness.

The study involved three datasets obtained from the same population: (1) D_H - general inpatient cohort from a pre-pandemic dataset collected between January 2016 and December 2019, (2) D_W1- COVID-19 patients from the first wave of the pandemic enrolled between March and July 2020, and (3) D_W2- COVID-19 patients from the second wave of the pandemic enrolled between August 2020 and June 2021.

The observations corresponding to each patient were characterized as per a 77-dimensional feature vector along with a label displaying a respiratory deterioration event within 24 hours. The features comprised commonly assessed laboratory parameters, vital signs, and variance over time.

The experimental process utilized two task definitions that included patient deterioration prediction tasks. The first task was a respiratory deterioration prediction task T_RD which was based on the increase in the level of oxygen support required from level zero or one to level two or three or unplanned intensive care unit (ICU) admission. The second task was a general deterioration prediction task T_GD defined as either the composite mortality outcome or ICU admission.

The experimental setup was based on three scenarios: A, B, and C. In scenario A, the team demonstrated the value of employing historical data while pretraining RapiD_AI models with the background of a pandemic caused by a novel disease. D_H was used to pre-train deep learning models, D_W1 was used to either train the benchmark neural networks or XGBoost models or fine-tune pre-trained networks, and D_W2 was used as a held-out test dataset to assess model performance.  

Scenario B hypothesized that selecting the most relevant pretraining models could facilitate the achievement of superior performance compared to retraining all historical data. This would also reduce the computational requirement of the pretraining process. The team considered historical examples that had major similarities to COVID-19 data.

Pretraining samples were selected by using human experience in identifying five different disease clusters having varying degrees of similarity to the clinical pattern observed for COVID-19 and using a computational approach that utilized tSNE to cluster all historical data with COVID-19 samples obtained in the initial three weeks of the pandemic. Additionally, scenario C replicated the scenario of a healthcare system that was facing a pandemic and had access to deploy machine learning models.     

Results

The study results showed that the pretraining deep neural network (DNN) models from scenario A improved their performance during the initial 20 weeks of the COVID-19 pandemic. Pretraining these DNN models improved performance by 110.87% relative and 41.71% absolute AUC in the first week and a 3.86% of absolute average AUC in the following 19 weeks of the pandemic.

Furthermore, the RapiD_AI outperformed the baseline XGBoost model in the initial four weeks of the pandemic by 4.37% relative and 3.58% absolute AUC and the overall average by 4.92% relative and 4.21% absolute AUC. These performance improvements can translate into remarkable operational and clinical benefits in the context of a global pandemic. The average gain of 4.21% in the algorithm AUC implied an increase of up to 1399 additional accurate classifications weekly in the UK, which could further lead to improved patient medical interventions.

Scenario B identified that the most frequently noted International Classification of Diseases-10 (ICD10) codes from the 10% of the most similar clusters were I10, Z922, Z864, Z501, I489, N179, Z867, Z921, E119, N390. However, the team highlighted that the code frequency in the general population could impact the composition of the frequently occurring ICD10 codes from selected training clusters and that multiple ICD10 codes observed for every patient made it difficult to ascertain the primary diagnosis.

Scenario C resulted in an 11.93% relative and a 9.32% absolute AUC improvement in performance over the initial two weeks of the pandemic compared to the XGBoost dataset training on only weekly information. The performance gain was consistent over the initial 20 weeks, with an average relative and absolute AUC increase of 7.57% and 6.42%, respectively.

Conclusion

Overall, the study highlighted the working of the RapiD_AI framework as a tool for pandemic preparedness along with the usefulness of machine learning during a pandemic.    

*Important notice: medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.