ChatGPT versus expert feedback on clinical reasoning questions and their effect on learning: a randomized controlled trial
Abstract
PURPOSE
To evaluate the effectiveness of ChatGPT-generated feedback compared to expert-written feedback in improving clinical reasoning skills among first-year medical students.
METHODS
This is a randomized controlled trial conducted at a single medical school and involved 129 first-year medical students who were randomly assigned to two groups. Both groups completed three formative tests with feedback on urinary tract infections (UTIs; uncomplicated, complicated, pyelonephritis) over five consecutive days as a spaced repetition, receiving either expert-written feedback (control, n = 65) or ChatGPT-generated feedback (experiment, n = 64). Clinical reasoning skills were assessed using Key-Features Questions (KFQs) immediately after the intervention and 10 days later. Students' critical approach to artificial intelligence (AI) was also measured before and after disclosing the AI involvement in feedback generation.
RESULTS
There was no significant difference between the mean scores of the control (immediate: 78.5 ± 20.6 delayed: 78.0 ± 21.2) and experiment (immediate: 74.7 ± 15.1, delayed: 76.0 ± 14.5) groups in overall performance on Key-Features Questions (out of 120 points) immediately (P = .26) or after 10 days (P = .57), with small effect sizes. However, the control group outperformed the ChatGPT group in complicated urinary tract infection cases (P < .001). The experiment group showed a significantly higher critical approach to AI after disclosing, with medium-large…
How strong is the evidence?
Evidence Hub reads this as a randomized trial, done in people. Its evidence level is 2 of 5 (High · randomized trial). Randomized trials of a reasonable size are the strongest single-study design. Reported size: 65 participants or records.
Read the paper
Related studies
- Improving the translation of search strategies using the Polyglot Search Translator: a randomized controlled trial
- Nutritional, Physical, Cognitive, and Combination Interventions and Frailty Reversal Among Older Adults: A Randomized Controlled Trial
- Clinical characteristics and risk factors associated with COVID-19 severity in patients with haematological malignancies in Italy: a retrospective, multicentre, cohort study
- A randomized, controlled trial of oral propranolol in infantile hemangioma
- Clinical phenotypes of delirium during critical illness and severity of subsequent long-term cognitive impairment: a prospective cohort study
- Increased physical activity improves sleep and mood outcomes in inactive people with insomnia: a randomized controlled trial
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.