Investigating retrieval-augmented code generation for domain-specific tasks
Abstract
In recent years, Large Language Models (LLMs) have increasingly influenced both everyday life and a wide range of research domains, attracting considerable interest from the research community. Although previous studies have shown that LLMs trained specifically on code can successfully solve a variety of programming tasks, including code completion, code generation, and code translation, they can still face essential limitations coming from their reliance on static training data. In particular, most models lack access to up-to-date, domain or project-specific information at inference time, leading to incorrect or outdated outputs. Retrieval-Augmented Generation (RAG) has been proposed as an effective approach to address this limitation by incorporating relevant external knowledge. While RAG has demonstrated promising outcomes, its effectiveness for domain-specific code generation remains to be further investigated. This thesis evaluates the impact of retrieval-augmented code generation compared to standard code generation without retrieval across multiple domain-specific programming tasks. A general experimental framework is introduced to standardize different domain-specific benchmarks. To have a fair comparison, identical settings for the models and the generation are used. The evaluation is conducted on three distinct domains: quantum, cryptography, and game development. The generated code is evaluated using execution-based metrics (pass@k) when unit tests are available…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.