Bidirectional Generation of Behavior-Driven Development Artifacts Using Fine-Tuning and Retrieval-Augmented Generation
Abstract
This thesis investigates bidirectional generation between semi-structured records and structured Behavior-Driven Development (BDD) scenarios using Large Language Models (LLMs). It compares fine-tuning (CodeT5+ and Qwen2.5-Coder) with Retrieval-Augmented Generation (RAG) (Meta-Llama-3.1-8B, DeepSeek-Coder) across a curated dataset of 2,100 aligned pairs. Results demonstrate that generation quality is heavily influenced by task direction and context management. Fine-tuned models significantly outperformed RAG-based methods, achieving F1 scores up to 0.9549 and Exact Match scores of 0.8119. RAG struggled to maintain the strict structural fidelity required for Gherkin-based BDD artifacts, failing to achieve exact matches. These findings suggest that fine-tuning remains the most reliable strategy for generating technically accurate BDD artifacts, as it better handles the rigid structural constraints of automated testing specifications compared to in-context prompting alone.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.