Diversity-Aware Document Retrieval using Retrieval-Augmented Generation
Abstract
Retrieval-Augmented Generation (RAG) systems rely on retrieved context to support grounded answer generation. However, standard nearest-neighbor retrieval may return chunks that are individually relevant to a query but collectively redundant or concentrated in a small number of source documents. This can limit evidence coverage and reduce the effective use of the language model's context budget, especially when retrieval is performed over long scientific documents. This thesis develops and evaluates a document-based RAG pipeline with a focus on diversity-aware retrieval. The pipeline uses a two-stage structure: Stage 1 retrieves candidate documents, and Stage 2 selects fine-grained body chunks as evidence for answer generation. For Stage 1, the thesis compares direct document representation retrieval with front-chunk aggregation. For Stage 2, it compares Relevance-Only Dense Retrieval with two diversity-aware selection strategies: Jaccard-Based Lexical Diversity Reranking and Embedding-Space Diverse Neighbor Selection. The system is evaluated on a corpus of scientific PDF documents using automatic proxy evaluation, manually written conceptual queries, retrieval-side context indicators, LLM-based context judging, and qualitative answer examples. The Stage 1 results show that direct document representation provides comparable candidate recall with lower retrieval time, while front-chunk aggregation can promote documents with stronger field-level matches. The Stage 2 results…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.