Evaluating the Performance of AI Large Language Models in Detecting Pediatric Medication Errors Across Languages: A Comparative Study
Abstract
Objectives: This study aimed to evaluate the performance of four AI models, (GPT-5, GPT-4, Microsoft Copilot, and Google Gemini), in detecting medication errors through pediatric case scenarios. Methods: A total of 60 pediatric cases were analyzed for the presence of medication errors, of which only half contained errors. The cases covered four therapeutic systems (respiratory, endocrine, neurology, and infectious). The four models were exposed to the cases in both English and Arabic using a unified prompt. The responses for each model were used to calculate various performance metric cover accuracy, sensitivity, specificity and reproducibility. Analysis was carried out using SPSS version 22. Results: Microsoft Copilot demonstrated relatively higher accuracy (86.7% in English, 85.0% in Arabic) compared to other models in this dataset, followed by GPT-5 (81.7% in English, 75.0% in Arabic). GPT-4 and Google Gemini had less accuracy, with Gemini having the lowest accuracy across all languages (76.7% in English, and 73.3% in Arabic). Microsoft Copilot showed comparatively higher sensitivity and specificity, particularly in cases of respiratory and infectious diseases. The accuracy in Arabic was lower compared to that of English for the majority of models. Microsoft Copilot exhibited relatively higher reproducibility and inter-run agreement (Cohen's Kappa = 0.836 English, 0.815 Arabic, p p Conclusions: In our evaluation, Microsoft Copilot demonstrated relatively higher…
Read the paper
Related studies
- Structural and topological analysis of thiosemicarbazone-based metal complexes: computational and experimental study of bacterial biofilm inhibition and antioxidant activity
- Toward better cures for Mycobacterium abscessus lung disease
- Challenges for global antibiotic regimen planning and establishing antimicrobial resistance targets: implications for the WHO Essential Medicines List and AWaRe antibiotic book dosing
- Design, Synthesis and In vitro Antitubercular Effect of New Chalcone Derivatives Coupled with 1,2,3‐Triazoles: A Computational Docking Techniques
- Upregulation of pmrA, pmrB, pmrC, phoQ, phoP, and arnT genes contributing to resistance to colistin in superbug Klebsiella pneumoniae isolates from human clinical samples in Tehran, Iran
- Whole-genome sequencing of clinical isolates of Citrobacter Europaeus in China carrying blaOXA−48 and blaNDM−1
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.