Using machine learning to shed light on how well the triage systems work

A multinational collaboration at Eitri medical innovation center in Bergen, Norway, has used machine learning models to identify patient groups at risk of being mistreated.

Doctors must make hard choices every day at the emergency department (often called the ED or ER). They usually have a system that helps them get the right level of care for patients, called triage.

Triage systems help doctors determine when a patient is facing a true medical emergency and need treatment immediately, or when it is probably safe to delay treatment. Still, mistakes can happen, and sometimes patients don't get the help they need."

Dr. Sage Wyatt at the Faculty of Medicine, University of Bergen, Norway

There is no universal triage system, and it can be difficult to evaluate how well different systems function.

Now, an eclectic team of Norwegian and German researchers from academic, industry, and medical sectors, has used machine learning to shed light on how well the triage systems work in Bergen and Trondheim, Norway.

The collaboration sprung out of a "Datathon" at Eitri Medical Incubator, Bergen in 2022, supervised by Dr. Christopher Sauer, an oncologist from the Institute for AI in Medicine from Essen, Germany. This project eventually led to a publication in the
Journal of Medical Internet Research.

First author, Dr. Sage Wyatt, defends the value of using machine learning to analyze medical data:

"In this setting, machine learning methods allowed us to consider the importance of many complex factors that impact triage classification simultaneously, providing more nuanced results than conventional methods", she says.

The results were surprising

The data scientists in Wyatt's team wanted to find out what kinds of patients were failed by the triage system, such as patients who were given low priority but who later died or got transferred to the intensive care unit (undertriage) or patients who were given high priority unnecessarily (overtriage).

Luckily, incorrect triage was very rare, affecting less than one percent of the patients.

The data science team members with industry training used machine learning models to determine the importance of many different patient characteristics simultaneously, a novel method that has not been used in this setting before.

The machine learning models ranked importance of variables on the dataset by a metric known as "SHAP-values."

"SHAP is a metric from game theory and is a way to quantify how much a variable contributes to prediction", explains Wyatt.

When they applied these methods to emergency department data, the results were surprising:

"A previous study using conventional methods in Bergen showed that overtriage was most prevalent in patients younger than 18 years, implying that age is one of the most important variables contributing to overtriage" Wyatt says

"Yet the automated variable selection based on SHAP-values in our study revealed that age might not be the most important characteristic contributing to overtriage in Bergen, as might have been an assumption in a study directed by domain knowledge."

Correcting assumptions

By "domain knowledge" Wyatt refers to the knowledge derived from the expert opinion of doctors who use the system. Wyatt says that it might have downsides to base a study on expert opinion alone:

"While it's valuable to hear opinions from people who know the triage systems best, this information can also be based on attitudes and assumptions unrelated to the tool itself", says Wyatt, and continues:

"With a lot of focus in society on gender equality, which is an important topic to discuss, doctors and patients might assume that incorrect triage could also be due to gender bias", she says.

The earlier study found that patient gender was an important factor, however, as Dr. Wyatt explains:
"Instead, with machine learning methods, we identified that the clinical referral department and diagnostic codes are more important factors associated with overtriage in the Bergen dataset."

"AI can provide new perspectives in science"

While machine learning methods and artificial intelligence aren't perfect tools, Wyatt hopes that these tools can be used to provide new perspectives in science.

"For optimal usage, appropriate methods must be tailored to the specific research context, and common pitfalls need to be avoided," she says, and concludes:

"More research is needed in the future about triage systems and new applications of machine learning methods, such as automated triage classification systems".