New tool detects hidden biases in medical training data

Researchers have built a tool to uncover hidden issues in the massive datasets used to train medical AI.

The tool searches training data for subtle patterns that could lead AI models to incorrect conclusions, potentially jeopardizing patient care. The work aims to help researchers and regulators make artificial intelligence more reliable and trustworthy for real-world clinical use.

The models that drive precision medicine learn to infer clinical outcomes from the data they're trained on. In many cases that works great but it can also lead to interesting failures that aren't immediately apparent. The tool we developed gives you a clear understanding of which elements of metadata are posing the most risks for your model to pick up a bias."

Mathias Unberath, senior author, expert in AI assisted medicine, Johns Hopkins University

The work, a collaboration between Johns Hopkins and the U.S. Food and Drug Administration, is newly published in npj Digital Medicine.

The "Clever Hans Phenomenon" refers to a horse named Hans that in the early 1900s got famous for being able to do math, tapping out correct answers with a hoof. Turns out, Hans couldn't do math but was quite good at reading his human's body language that inadvertently revealed how he should respond.

Medical AI models can be a lot like Hans, Unberath said, reaching conclusions by reading the wrong signals.

"Models take the same shortcuts," he said, pointing to AI trained to accurately predict a person's sex by looking at their eyes. Turns out the model was relying on the presence of mascara rather than any biological features.

"We train models to be the most predictive, but we have zero control over what the model uses to make the prediction."

The team's tool detects the potential for such flawed associations.

While others have built safeguards based on how an AI model performs, this new tool, called Generalized Attribute Utility and Detectability-Induced bias Testing, or G-AUDIT, examines the training data itself for signs that a model could learn from unintended cues rather than clinically meaningful signals.

"The most exciting thing is how this is a shift in how we do the work," said co-author Mitchell Pavlak, a PhD student in Unberath's lab at Johns Hopkins. "We're not checking our work after the fact. We analyze the data to figure out what is likely to be a problem."

G-AUDIT identifies and ranks attributes in the data likely to lead the AI to make potentially incorrect conclusions.

Researchers tested it on a range of medical data including images, text, and spreadsheets. It revealed hidden flaws in data collection, imaging conditions and other elements likely to lead to flawed conclusions about a patient's health.

For instance, a skin cancer dataset included images from a cancer clinic that served a high-risk population, and images from a general dermatology clinic that saw fewer cancer cases. Because the imaging systems the clinics had varied in quality, an AI trained on the dataset could wrongly deduce that camera quality is a key predictor of cancer. Or even that the presence of rulers in images predicts cancer because the cancer clinic tended to include rulers in images to track the size of growths over time.

"Envision taking this algorithm into the real world with a smart phone camera," Unberath said. "There's no ruler. The camera quality is completely irrelevant. But the model learned to associate 'ruler' and 'camera.' You have just created health disparity."

The team plans to continue to build out and expand the tool, which they say could be adapted to use beyond healthcare.

"The core takeaway is that right now people are auditing models but they don't really know what to audit for. We are changing that," Unberath said. "If you only audit for things that are top of mind to you, you're probably missing a lot of things that could lead to spurious correlations."