Utvärdering av Retrieval-Augmented Generation och finjustering för automatiserad generering av Python-testfall
Abstract
Unit tests are vital to software development to prevent malfunctioning code to reach production, ensuring this event does not occur is incredibly important; however, creating test cases is time consuming and prone to human errors. Artificial intelligence could automate this process and ensure high-quality tests, minimizing risks of failure. However, large language models still lack the domain-specific knowledge to replace humans in this task. This thesis aims to evaluate two different methods, namely fine-tuning and Retrieval-Augmented Generation (RAG), to improve the performance of these LLMs within test case generation. Both a fine-tuned and an RAG model were developed, using the codellama:7b-instruct model as the base model. The models were trained using an open-source dataset grouping together test cases with their focal methods to improve the models’ knowledge of test case generation capabilities. The three models were then used to generate test cases for a set of methods to evaluate whether there was any improvement in test case generation performance. The results showed that RAG underperformed just below the base model’s capacity, while the fine-tuned model performed poorly overall. This may be due to a lack of context for complex method relationships, or due to an overall limitation for the codellama:7b-instruct model’s capabilities. Finally, it is proposed that further research in automatic test case generation for Python is needed, but that LLMs could be viewed as…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.