Evaluating Trade-offs Between Performance and Resource Utilization in Retrieval-Augmented Generation and Finetuning for Domain-Specific Language Models
Abstract
This report investigates the trade-offs between Retrieval Augmented Generation (RAG) and fine-tuned Language Models for a domain-specific question answering task. A high-performing large language model (LLM) delivers practical value only when it can be deployed within the constraints of the target environment. We focus on NASA technical reports to generate a dataset, a perfect fit for jargon-rich domain. First, we extract the content of the PDFs and generate corresponding questions. Secondly, we conduct experiments using simple RAG architecture and LLMs finetuned exclusively on our dataset. Lastly, we evaluate both the approaches based on the standard performance metrics as well as the computation resources consumed to produce the associated results. The performance analysis aspect of our research highlights the strength and weaknesses of each method in handling domain queries. Whereas resource monitoring ensure that the techniques are efficient and scalable enough to be utilized in real-life practices given the resource constraint environments. Based on the results, we recommend the more effect approach while providing practical guidelines for choosing retrieval-based and finetuning-based strategies, balancing performance with the available resources.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.