A comparative study of retrieval-augmented generation and fine-tuned language models for domain-specific QA
Abstract
Adapting large language models to domains they were never trained on is an increasingly relevant challenge. This issue is particularly difficult when working with self-contained closed systems or environments, such as those based upon fictional worlds, that include complete access to relevant knowledge contained within a single textual data source, which does not appear in any general pretraining corpus. There are two prominent approaches to tackle this task: fine-tuning and retrieval-augmented generation. The former updates the model parameters on the domain text, while the latter supplies relevant passages at inference time for the model. While both have been well-researched individually, their direct comparison with a strictly closed domain, particularly with unsupervised training, remains scarce. This thesis compares unsupervised LoRA fine-tuning and RAG for domain-specific factual question answering, using the fantasy novel Fourth Wing as a controlled domain corpus. A manually curated dataset of 100 question–answer pairs is used to evaluate three small instruction-tuned models, Gemma-2-2b-it, Phi-3.5-mini-instruct, and Qwen2.5-3B-Instruct, across baseline, RAG, and fine-tuned conditions. Performance is assessed through both manual and automatic metrics, more specifically manual accuracy annotation, SBERT cosine similarity, ROUGE-L, Exact Match, and inference latency. The results show that RAG consistently outperformed fine-tuning. Manual accuracy reached up to 67% under…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.