Retrieval augmented generation for question-answering systems
Abstract
To enhance the factual accuracy and reliability of large language models (LLMs) during the question-answering process, researchers typically employ a framework called Retrieval Augmented Generation. Current RAG methods mainly rely on injecting retrieved documents into the input context of LLMs; this method is called in-context learning (ICL). Although effective, ICL-based RAG suffers from increased computational overhead with longer contexts and a fundamental gap between how external knowledge is presented (in the input) and how internal knowledge is stored (in parameters). Graph-based RAG methods, such as GraphRAG and HiRAG, attempt to improve retrieval quality by leveraging knowledge graph structures but introduce significant complexity in knowledge graph construction, requiring substantial computational resources and time. This dissertation investigates Parametric Retrieval-Augmented Generation (PRAG), a novel approach that leverages Low-Rank Adaptation (LoRA) to incorporate external knowledge directly into the feed-forward network parameters of large language models. It does not require additional context in the input to incorporate retrieved information; its overall architecture is also simpler than GraphRAG, and compared to ICL-based RAG, it generates higher-quality responses with shorter inference time. The PRAG framework is successfully reproduced across three LLM backbones and four benchmark datasets, with results closely matching the original paper. During the…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.