Retrieval-Augmented Generation on PIRLS Documents - A comparative evaluation of baseline and agentic pipelines using open-weight LLMs and DeepEval metrics
Abstract
Researchers and other knowledge workers are using AI increasingly to find information faster, but general-purpose tools often fall short in professional settings. Closed commercial models can confidently return incorrect information, and they usually cannot “see” an organization’s internal documents—so they are not very helpful when the question depends on proprietary knowledge. On top of that, cloud-based AI tools can create security concerns, since sensitive data are sent to external servers over the internet. To address these issues, this project proposes a prototype Information Retrieval (IR) system that combines Retrieval-Augmented Generation (RAG) with open-source large language models (LLMs) running entirely on local hardware. The goal is to show, through experimentation, that RAG can let an LLM safely use internal knowledge bases to produce accurate, context-aware answers—without giving up control of the data. To evaluate the prototype, we use documents from Progress in International Reading Literacy Study (PIRLS), selected because they are text-heavy and domain-specific. System outputs are evaluated with an LLM-as-a-judge framework (DeepEval) across four metrics—Context Precision, Context Recall, Faithfulness, and Answer Correctness—and we measure end-to-end response time to assess feasibility for interactive use on local hardware (see Section 4.4.1). The study compares two approaches: a basic (naive) RAG pipeline and an advanced (agentic) pipeline that adds query…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.