Five leading AI models faced the same 227 infectious disease exam questions, testing whether their accuracy could withstand the demands of specialized knowledge, complex clinical problems, and visual interpretation.
Study: Performance of five large language models (LLM) on infectious diseases related questions. Image Credit: Jack_the_sparow / Shutterstock
In a recent study published in the journal Scientific Reports, researchers at the University of the Pacific School of Pharmacy, Stockton, USA, compared the accuracy of five large language models (LLMs) in answering infectious disease-related pharmacy examination questions across different pharmacy subject areas, cognitive levels, and question formats.
Background
Can artificial intelligence (AI) reliably answer questions that future healthcare professionals must master? LLMs have the potential to be educational tools that generate explanations and answers to support interactive learning. Their use in healthcare may also raise ethical concerns. Incorrect information may affect students' and educators' understanding of clinical concepts and potentially influence clinical judgment. Previous studies evaluating LLMs in the discipline of pharmacy education showed inconsistent performance. LLMs may raise concerns about academic integrity and the development of critical thinking skills. Further research is needed to establish their reliability across pharmacy subjects and cognitive levels.
About the study
The researchers conducted a comparative analysis of five commercially available LLMs using examination questions from an Integrated Clinical Sciences Infectious Diseases course, a required second-year course in the accelerated three-year Doctor of Pharmacy program at the University of the Pacific School of Pharmacy. The course ran from January to April 2025, and data were collected from May to July 2025.
The evaluated models were ChatGPT-4.1 and ChatGPT-4o-mini (OpenAI), Claude Sonnet 4 (Anthropic), Copilot Web/Mobile/Edge (Microsoft), and Gemini 2.5 Flash (Google). The researchers evaluated 227 examination questions drawn from the course, covering medicinal chemistry, clinical microbiology, pharmacology, therapeutics, and pharmacokinetics and pharmacodynamics. Bloom's taxonomy was used to classify questions into remember/understand, apply/analyze, and evaluate/create categories. The questions were also categorized by format (multiple-choice or fill-in-the-blank) and whether they included images.
Each model received identical prompts and instructions to answer the questions and explain why the selected answer was correct and the alternative options were incorrect. Instructor-provided answer keys were used to evaluate the responses, which were then classified as correct or incorrect. The supporting explanations were not separately graded for completeness or clinical safety. Two investigators independently reviewed questions that any model answered incorrectly and agreed on all classifications. Descriptive statistics were calculated to analyze model accuracy. The chi-square test was used to compare accuracy across models and question categories. Cohen's kappa coefficient was used to evaluate agreement between models. Statistical significance was set at p < 0.05, and the data were analyzed in Microsoft Excel (Microsoft 365).
Study results
The evaluation included 227 questions: 108 on therapeutics, 45 on medicinal chemistry, 40 on clinical microbiology, 25 on pharmacology, and 9 on pharmacokinetics/pharmacodynamics. Based on Bloom's taxonomy, 92 questions assessed remembering and understanding, 71 assessed applying and analyzing, and 64 assessed evaluating and creating. Most questions (212; 93.4%) were multiple-choice, 15 (6.6%) were fill-in-the-blank, and 13 questions (5.7%) contained images.
Across all five models, the overall accuracy was 90.2%. ChatGPT-4o-mini achieved the highest accuracy (91.2%), followed by Gemini 2.5 Flash (90.7%) and ChatGPT-4.1 (90.3%). Copilot and Claude recorded the lowest accuracy, at 89.4% each. The difference between the highest- and lowest-performing models was only 1.8 percentage points, indicating relatively similar performance across the evaluated systems.
Accuracy varied across subject areas. Pharmacology had the highest accuracy at 93.6%, followed by therapeutics at 93.0%, pharmacokinetics/pharmacodynamics at 91.1%, clinical microbiology at 87.5%, and medicinal chemistry at 84.0%. Compared with therapeutics, medicinal chemistry and clinical microbiology showed significantly lower accuracy (p < 0.05). Differences relative to therapeutics were not statistically significant for pharmacology (p = 0.80) or pharmacokinetics/pharmacodynamics (p = 0.64). These results suggest that model accuracy depends partly on the pharmacy subject being assessed.
Performance also varied according to cognitive complexity. Accuracy was 92.0% for remember/understand questions and 92.1% for apply/analyze questions, with no significant difference between these categories (p = 0.93). By comparison, accuracy declined to 85.6% for evaluate/create questions, representing a statistically significant reduction compared with remember/understand questions (p less than 0.05). Claude demonstrated particularly low accuracy in the evaluate/create category, at 79.69%.
No statistically significant difference was observed in model performance when comparing multiple-choice and fill-in-the-blank questions. Only 15 questions used the latter format, limiting this comparison. Image inclusion was associated with a substantial decline in accuracy. The models correctly answered 45 of 65 responses to image-containing questions (69.2% accuracy), compared with 979 of 1,070 responses to text-only questions (91.5% accuracy; p < 0.05). Eight of the 13 image-containing questions included antimicrobial chemical structures, which may have contributed to the lower performance. The small image sample and its chemical content limit the ability to draw broader conclusions about visual interpretation.
Inter-model agreement, assessed using Cohen's kappa coefficient, ranged from 0.497 to 0.613, indicating predominantly moderate agreement overall, with the strongest pairing reaching the authors' good-agreement category. ChatGPT-4.1 and Copilot demonstrated the highest agreement (κ = 0.613), whereas ChatGPT-4o-mini showed the lowest agreement with both Claude and Copilot (κ = 0.497). The findings indicated that similar overall accuracy did not necessarily mean that the models answered the same individual questions correctly.
The findings have implications for pharmacy education. They suggested that students could use LLMs to generate explanations and practice questions, while comparing responses from different models could help develop critical evaluation skills. The authors advised students to verify AI-generated information against reliable references. The authors discussed concerns regarding academic integrity, cognitive offloading, and the possible effects of excessive reliance on AI during learning. These were educational concerns discussed by the authors, rather than outcomes directly measured in the study.
Conclusions
The study demonstrated that five evaluated LLMs achieved high overall accuracy when answering infectious disease-related pharmacy examination questions, with relatively minor differences between models. Performance varied across subject areas and cognitive levels, with lower accuracy in medicinal chemistry, clinical microbiology, higher-order reasoning, and image-based questions. Moderate agreement between models further highlighted the need to independently verify AI-generated responses rather than relying on a single system. Accuracy on examination questions does not establish the reliability or safety of clinical recommendations. The findings support the potential use of LLMs as supplementary educational resources, provided that faculty offer appropriate guidance.
The study was limited to questions from one pharmacy program, with uneven subject coverage and only 13 image-containing questions. Question wording may affect model responses, and the results reflect testing from May to July 2025 rather than later model updates. Further educational research should examine effective AI integration while minimizing misinformation and excessive dependence on these technologies.
Source:
Journal reference:
High e-health literacy encourages patients to visit medical professionals