Local Deployment of Large Language Models for AI-Assisted Software Development : Evaluating Quantization, Retrieval-Augmented Generation and LoRA for IDE-Based Developer Support
Abstract
AI-assisted developer tools are increasingly used in software engineering, but many existing solutions rely on cloud-based inference, which raises concerns related to privacy, source-code exposure, latency, cost and organizational control. This thesis investigates whether open-source large language models can be deployed locally as practical developer assistants in an industrial software development environment. A local prototype was implemented using an IDE client, a FastAPI-based model server, local inference backends, RAG, quantized models and LoRA-based fine-tuning. The system was evaluated through controlled benchmark experiments, latency measurements and system-level observations from prototype implementation. The experiments compared model size, quantization, retrieval depth and parameter-efficient fine-tuning with respect to output quality, latency and deployment feasibility. The results suggest that locally deployed AI developer assistance can be feasible in enterprise environments, although long-term workflow integration remains outside the scope of this study. The 3B-Q4 model provided the strongest overall trade-off between benchmark score and latency, achieving performance close to the larger 7B-Q4 model while requiring substantially lower response time. Retrieval improved performance in selected cases, especially at moderate retrieval depth, but increased prompt size and latency. LoRA improved in-distribution behavior on repository-specific examples, but did not…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.