Study shows AI tool's high accuracy in answering genetic counseling questions

NewsGuard 100/100 Score

An artificial intelligence tool correctly answered 83 percent of common genetic counseling questions, including those about genetic testing and genetic syndromes, a new study found.

Presented at the Society of Gynecologic Oncology's Annual Meeting on Women's Cancer on March 17, 2024, in San Diego, the study examined the capabilities of a form of artificial intelligence (AI) called generative AI. Such tools predict likely options for the next word in any sentence based on how billions of people used words in context on the internet. A side effect of this next-word prediction is that generative AI chatbots like ChatGPT can generate replies to questions in realistic language and produce clear summaries of complex texts.

Led by researchers at NYU Langone Health and its Perlmutter Cancer Center, the current paper explored the application of ChatGPT to answering questions about genetic syndromes counseling related to gynecologic oncology.

Our data suggest that this tool has the potential to answer common questions from patients to reduce anxiety and keep them informed. More data input from gynecologic oncologists is needed before the tool can help to educate patients on their cancers, and only as an assistant to human providers."

Jharna M. Patel, MD, lead study investigator

Jharna M. Patel has a Gynecologic Oncology Fellowship at NYU Langone.

40 questions

For the study, the research team consulted with gynecologic oncologists to choose 40 questions often asked by patients and in line with professional society guidelines. They then typed the questions into prompted ChatGPT version 3.4, asking for answers.

The authors then asked attending gynecologic oncologists to rate the chatbot's answers on the following scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect, and 4) completely incorrect. The proportion of responses earning each score was calculated overall and within each question category.

Specifically, ChatGPT was found to have provided correct and comprehensive answers to 33/40 (82.5 percent) questions, correct but not comprehensive answers to 6/40 (15 percent) questions, and partially incorrect answers to 1/40 (2.5 percent) questions. None of the answers were scored as completely incorrect.

The genetic counseling category of questions (e.g., How do you know if someone has an inherited or family cancer syndrome?) had the highest proportion of AI answers that were both correct and comprehensive, with ChatGPT answering all 20 questions with 100 percent accuracy. ChatGPT performed equally well with questions about specific genetic disorders. The gynecologic oncologists found, for instance, that 88.2 percent (15/17) of the answers were correct and complete on testing for differences in the BRCA1 or BRCA2 gene known to be a driver of cancer risk, while 66.6 percent (2/3) were correct on Lynch syndrome, the most common form of hereditary colorectal cancer. A complete list of the study questions is included below.

"We think we can further improve on these results by continuing to train the AI tool on more data, and by learning to ask better sets of questions," says senior study author Marina Stasenko, MD, an assistant professor in the Department of Obstetrics and Gynecology at NYU Grossman School of Medicine. "The goal is to deploy this in the clinic when ready, but only as an assistant to human providers."

Along with Dr. Patel and Dr. Stasenko, study authors in the Department of Obstetrics and Gynecology's Division of Gynecologic Oncology were Catherine Hermann, MD, and Whitfield B. Growdon, MD. Study author Emeline M. Aviki, MD, MBA, was from the Division of Gynecologic Oncology in the Department of Obstetrics and Gynecology at NYU Grossman Long Island School of Medicine.

Comments

The opinions expressed here are the views of the writer and do not necessarily reflect the views and opinions of News Medical.
Post a new comment
Post

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions.

You might also like...
AI models challenge humans in understanding minds, but struggle with subtleties, study finds