Comparing Retrieval Strategies in a Domain-SpecificRetrieval-Augmented Generation System
Abstract
Retrieval-Augmented Generation (RAG) systems areincreasingly used to improve the accuracy of large languagemodels and reduce hallucinations by linking generation toexternal knowledge sources. However, most current studiesfocus on large knowledge bases and general domains, whilethe impact of different retrieval strategies within small-scale,domain-specific RAG systems remains insufficientlyunderstood.This study aims to evaluate the impact of three differentretrieval strategies TF-IDF, BM25, and Dense Retrievalwithin a small-scale RAG system dedicated to softwareengineering. It also aims to compare these systems with abaseline that relies on passing the entire corpus directly tothe language model without retrieval, to determine whetherretrieval is still necessary in small environments.A custom knowledge base was created, consisting ofapproximately 300 documents related to softwareengineering topics such as GitHub, Stack Overflow API,CI/CD, and OAuth. In addition, a set of questions andreference data was created, including direct, indirect, andrephrased questions.The RAG system was implemented using a native languagemodel via Ollama, and the systems were evaluated usingPrecision@K, Recall@K, and NDCG metrics to assess retrievalquality, as well as ROUGE-L and BERTScore to assess thequality of generated responses, with response time measuredfor each system.The results showed that Dense Retrieval performed bestacross most metrics, including retrieval quality and responsequality.…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.