Differences between human and AI scoring: A meta-analysis of english language assessments
Abstract
Despite a burgeoning body of research on using AI scoring systems in English assessments, concerns regarding their reliability persist. To fill this gap, this meta-analysis examined the AI-human scoring differences and the variables moderating these differences by synthesizing the results of 21 empirical studies with a total of 401,698 participants. Results indicate no statistically significant differences between AI and human scoring; the small effect size implies that the average systematic difference between the two was relatively modest. However, extremely high heterogeneity suggests that this overall finding masks considerable variability across study conditions. Moderator analyses reveal that AI-human scoring differences are significantly influenced by factors such as AI system type, number of human raters, agreement index employed, learner proficiency level (CEFR), and publication year. These findings suggest that while AI cannot fully replace human judgment, it can serve as a diagnostic reference tool within broader quality assurance frameworks. When significant discrepancies arise, they warrant investigation of both scoring sources. Based on these findings, this study offers evidence-based recommendations for educators on the effective use of AI scoring systems in language assessments.
How strong is the evidence?
Evidence Hub reads this as a meta-analysis, done in people. Its evidence level is 1 of 5 (Highest · pooled evidence). Pooled evidence from several studies sits at the top of the evidence pyramid. Reported size: 401,698 participants or records.
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.