ICU risk scores can be repurposed as continuous markers of illness severity

Commonly used ICU risk scores can be "repurposed" as continuous markers of severity of illness in critically ill patients--providing ongoing updates on changes in the patient's condition and risk of death, according to a study in the March issue of Critical Care Medicine, official journal of the Society of Critical Care Medicine (SCCM). The journal is published by Wolters Kluwer.

Three ICU risk scores, designed for different purposes, "performed reasonably well as a marker of severity of illness at admission, as well as throughout the ICU stay, demonstrating that it is possible for risk models to perform well even when deployed for uses other than originally intended," write Omar Badawi, PharmD, MPH, FCCM, and colleagues of Philips Healthcare, Baltimore. The study illustrates how the use of "Big Data" can advance our understanding of critically ill patients, drawing on extensive data contributed by tele-ICU programs across the United States.

One of Three ICU Risk Models Performs Best in Predicting Mortality

The study used data from the Philips eICU Research Institute database, which assembles clinical data from a diverse mix of hospitals using tele-ICU software. The analysis included de-identified data on more than 560,000 ICU patient stays, contributed by 333 ICUs at 208 US hospitals. The data covered nearly 39 million patient-hours of ICU care.

The analysis transformed three widely used ICU risk scores--designed to be assessed for different clinical scenarios--into continuous risk markers:

• Acute Physiology and Chronic Health Evaluation IV (APACHE): Perhaps the best-known ICU risk score, originally designed to estimate the risk of death on admission to the ICU.
• Sequential Organ Failure Assessment (SOFA): Originally designed to assess organ failure risk in patients with sepsis, the SOFA score was later validated for assessment every 48 hours to assess mortality risk.
• Discharge Readiness Score (DRS): Designed to assist ICU discharge decisions by estimating the risk of death in the first 48 hours after the patient leaves the ICU.

Hourly values for all three scores were evaluated as predictors of mortality risk. Overall, 4.8 percent of patients died in the ICU.

As continuous markers, all three models showed good accuracy in predicting the risk of death in the ICU, as well as risk of death within 24 hours. Of the three scores, the DRS had the highest predictive value. For patients in the highest ten percent of risk based on average score, the percentage who died was 30.4 percent with APACHE (average score > 80.0), 26.5 percent with SOFA (average score >7.6), and 36.4 percent with DRS (average score >13.0). The results were similar on analysis of hospitals with larger numbers of patients and deaths.

Looking closely at trends in each score during the ICU stay provided insights into the differences between the three models. Even in survivors, APACHE and SOFA scores tended to increase throughout the first 24 hours in the ICU. In contrast, the DRS decreased during the first 24 hours for survivors. That suggests that the DRS was more responsive to changes in patient condition--especially improvements.

The authors hypothesize that these differences may be attributed to model design. Both APACHE and SOFA primarily rely on the "worst" values of the preceding 24 hours to calculate a score. In contrast, DRS relies primarily on summary measures of the preceding 24 hours, such as averages and coefficient of variation.

Those observations highlight the need to be aware of differences in model design, and how the predictions may vary across ICU settings and in patients with varying characteristics. To help ICU clinicians understand these differences, Dr. Badawi and colleagues have created an online interactive data visualization tool.

"These findings suggest that it is possible to repurpose risk models for use outside of their original design, but with caveats," Dr. Badawi and coauthors conclude. They emphasize the need for further studies to determine the clinical relevance of continuous severity of illness indicators, and the need for caution when introducing these models in differing ICU settings. Dr. Badawi added, "Risk models provide objective data regarding patient status but users must combine their clinical judgment with an understanding of the model design in order to interpret the score for specific patients."