Developing a High-Quality Programming Question Bank Through Retrieval-Augmented Generation and Triadic Evaluation: An Empirical Study
Abstract
This study investigates how the integration of Retrieval-Augmented Generation (RAG) and a triadic evaluation mechanism can be leveraged to develop an automated system capable of generating high-quality programming questions, thereby reducing teachers' workload in question design and review. The system utilizes course materials and existing question banks as its knowledge base, generating new items through the AnythingLLM framework and conducting preliminary screening with a large language model (LLM). Adopting a mixed-methods research design, the study incorporates three evaluative perspectives-teachers, students, and AI-to examine pedagogical appropriateness, usability, and technical integrity. Experimental results indicate that, after iterative refinement, AI-generated questions achieved high ratings from both teachers and students in terms of curricular relevance and appropriate difficulty, with their evaluations showing a significant positive correlation $(r= 0.651, p<.05)$. The findings demonstrate that the triadic evaluation framework effectively calibrates the quality of AIgenerated questions and highlight the potential of AI as a supportive “teaching assistant” in programming education.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.