AI screening algorithms to diagnose diabetic retinopathy do not show consistent performance

Diabetes continues to be the leading cause of new cases of blindness among adults in the United States. But the current shortage of eye-care providers would make it impossible to keep up with demand to provide the requisite annual screenings for this population. A new study looks at the effectiveness of seven artificial intelligence-based screening algorithms to diagnose diabetic retinopathy, the most common diabetic eye disease leading to vision loss.

In a paper published Jan. 5 in Diabetes Care, researchers compared the algorithms against the diagnostic expertise of retina specialists. Five companies produced the tested algorithms - two in the United States (Eyenuk, Retina-AI Health), one in China (Airdoc), one in Portugal (Retmarker), and one in France (OphtAI).

The researchers deployed the algorithm-based technologies on retinal images from nearly 24,000 veterans who sought diabetic retinopathy screening at the Veterans Affairs Puget Sound Health Care System and the Atlanta VA Health Care System from 2006 to 2018.

The researchers found that the algorithms don't perform as well as they claim. Many of these companies are reporting excellent results in clinical studies. But their performance in a real-world setting was unknown. Researchers conducted a test in which the performance of each algorithm and the performance of the human screeners who work in the VA teleretinal screening system were all compared to the diagnoses that expert ophthalmologists gave when looking at the same images. Three of the algorithms performed reasonably well when compared to the physicians' diagnoses and one did worse. But only one algorithm performed as well as the human screeners in the test.

It's alarming that some of these algorithms are not performing consistently since they are being used somewhere in the world."

Aaron Lee, Lead Researcher, Assistant Professor of Ophthalmology, University of Washington School of Medicine

Differences in camera equipment and technique might be one explanation. Researchers said their trial shows how important it is for any practice that wants to use an AI screener to test it first and to follow the guidelines about how to properly obtain images of patients' eyes, because the algorithms are designed to work with a minimum quality of images.

The study also found that the algorithms' performance varied when analyzing images from patient populations in Seattle and Atlanta care settings. This was a surprising result and may indicate that the algorithms need to be trained with a wider variety of images.

Source:
Journal reference:

Lee, A.Y., et al. (2021) Multicenter, Head-to-Head, Real-World Validation Study of Seven Automated Artificial Intelligence Diabetic Retinopathy Screening Systems. Diabetes Care. doi.org/10.2337/dc20-1877.

Comments

The opinions expressed here are the views of the writer and do not necessarily reflect the views and opinions of News Medical.
Post a new comment
Post

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions.

You might also like...
Semaglutide linked to increased risk of severe eye condition NAION in new study