Evaluation of Arabic-Language AI Chatbot Responses to Migraine-Related Questions: A Comparative Cross-Sectional Study
Abstract
Background/Objectives: Migraine is a common and disabling neurological disorder, and many individuals increasingly seek information online. With the growing use of large language models (LLMs), such as ChatGPT, for patient education, concerns have emerged regarding the quality and reliability of the responses they generate, particularly in Arabic, where evidence remains limited. This study aimed to evaluate the reliability, quality, and accuracy of Arabic-language responses to frequently asked questions (FAQs) about migraine. Methods: A total of 25 FAQs were selected using a multisource approach and entered into four LLMs (ChatGPT-4.1, Gemini 3 Flash, DeepSeek-V3.2, and Grok 4.1), generating 100 responses. Responses were evaluated by a panel of expert neurologists using the modified DISCERN (mDISCERN), Global Quality Scale (GQS), and an accuracy scale. Inter-rater reliability was assessed using the intraclass correlation coefficient (ICC). Results: Significant differences were observed between chatbots for mDISCERN and GQS (both p p = 0.072). DeepSeek and Grok demonstrated the highest mDISCERN scores (34.07 ± 1.31 and 34.29 ± 2.59, respectively), while DeepSeek achieved the highest GQS (4.95 ± 0.13). The clearest between-model differences were observed in source transparency and communication of uncertainty. Inter-rater reliability was good across all instruments (ICC range, 0.799-0.831). Conclusions: Medical content generated by the chatbots was broadly comparable, whereas…
How strong is the evidence?
Evidence Hub reads this as a cross-sectional, done in people. Its evidence level is 4 of 5 (Low · observational). Observational designs can show an association but not prove cause.
Read the paper
Related studies
- Calcium Ions in the Physiology and Pathology of the Central Nervous System
- AI-powered in silico twins: redefining precision medicine through simulation, personalization, and predictive healthcare
- In silico functional, structural and pathogenicity analysis of missense single nucleotide polymorphisms in human MCM6 gene
- In Silico drug repurposing pipeline using deep learning and structure based approaches in epilepsy
- Targeting serotonin receptors with phytochemicals – an in-silico study
- In silico exploration of anti-prostate cancer compounds from differential expressed genes
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.