Label-Free Evaluation of Retrieval-Augmented Generation in a University Policy Chatbot
Abstract
Retrieval-augmented generation (RAG) is a common approach for building chatbots that combine fluent language generation with access to domain-specific knowledge. However, these systems often produce answers that are plausible but weakly supported by retrieved documents or that cite irrelevant sources. Evaluating RAG chatbots is difficult because it requires measuring both how well answers address user queries and how well they are grounded in retrieved content. University policy documents provide a useful test case. They are lengthy, formal, and regularly updated, yet they guide routine decisions for students and faculty. The combination of structured policies and varied user questions makes this domain well suited for systematic label-free evaluation. This study evaluates a RAG chatbot deployed at a large public university with over 700 pages of institutional policies. A dataset of 140 interaction logs was collected, and a refined subset was analyzed. Automated methods measured relevance (alignment of answers with questions), groundedness (alignment with retrieved text), and retrieval effectiveness (Recall@k and Mean Reciprocal Rank). Results show moderate performance on both relevance (0.64) and groundedness (0.66), with strong retrieval coverage but weaker ranking. Additional analyses examined how input clarity, phrasing style, and conversation behavior influenced chatbot performance.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.