Rethinking Automated Scoring in the Era of Artificial Intelligence: Emerging Directions for AI-Based Assessment
Abstract
The rapid development of generative artificial intelligence (AI), particularly large language models and multimodal models, is reshaping research on automated scoring in educational assessment. While previous research has often focused on the extent to which AI-generated scores agree with those assigned by human raters, agreement alone may not provide sufficient evidence for the quality or defensibility of an automated scoring system. This editorial argues that next-generation AI-based scoring research should move beyond model-level performance comparisons and examine the broader conditions under which AI-generated scores are produced, evaluated and used. It first discusses key design dimensions of AI-based scoring systems, including prompt design, rubric design, example-based calibration, model selection, scoring strategies and continuous performance monitoring. It then identifies emerging research directions involving evidence-grounded scoring, uncertainty and confidence calibration, process- and reasoning-aware evaluation, multimodal evidence, retrieval-augmented scoring, fairness and human–AI orchestration. Particular attention is given to iterative multi-agent and deliberative scoring, in which AI systems can review, critique and reconsider previous scoring decisions rather than independently producing a final score. The editorial concludes by proposing a broader research agenda in which automated scoring is evaluated not only in terms of agreement or accuracy, but also…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.