RagForge: Evaluating Retrieval-Augmented Generation for Long-Document Literary Question Answering
Abstract
RagForge is a retrieval-augmented generation (RAG) system built over the complete text of J.R.R. Tolkien's The Lord of the Rings trilogy to evaluate how RAG design choices affect grounded question answering on long narrative text. The system evaluates chunk size, retrieval depth, hybrid dense-BM25 weighting, generation parameters, query expansion, calibrated abstention, and generator model choice using a curated dataset of 1,000 factoid and multi-hop questions. Retrieval performance is evaluated using Hit@K, MRR, and nDCG, while generation quality is assessed using both keyword accuracy and LLM-as-a-judge scoring. The principal finding is that, once a strong retrieval baseline was established, generator capability represented a larger performance bottleneck than additional retrieval tuning. On identical retrieval with an 89.6% hit rate, Claude Opus 4.6 achieved 87.6% judge accuracy compared with 74.5% for Amazon Nova Pro, a 13.1 percentage-point difference. The study also finds that keyword-based evaluation can overstate RAG answer quality and that query expansion and score-based abstention did not improve performance on this domain-specific corpus.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.