Evaluation of a Retrieval Augmented Generation approach for technical documentation in manufacturing
Abstract
Recent advancements in Generative Artificial Intelligence (GenAI), driven by the development of Large Language Models (LLMs), have created opportunities for innovative solutions across multiple sectors, including manufacturing. However, deploying LLMs in industrial settings presents significant challenges due to their general-purpose nature, which often results in inaccurate or irrelevant responses when applied to domain-specific tasks. This paper explores the integration of Retrieval-Augmented Generation (RAG) to improve the performance of LLMs in industrial environments. The study investigates the effectiveness of five LLMs (GPT-4o, Mixtral-8x22B-Instruct-v0.1, Llama-3.3-70B-Instruct, DeepSeek-V3, Qwen2.5-72B-Instruct) in processing industrial technical documentation. The models’ performance was evaluated with and without RAG augmentation, using a dataset of 100 verified FAQs from an industrial machine manual. Evaluation criteria included expert evaluation, embedding-based evaluation, and comparative LLM assessment. Our findings show that RAG significantly improves the accuracy and relevance of LLM responses, achieving over 93% accuracy on general FAQs and more than 83% on specific, domain-dependent queries. The study demonstrates that RAG-based architectures provide a scalable, flexible solution to adapt LLMs to specialized manufacturing contexts. This research contributes to bridging the gap in understanding how to best implement LLMs in industrial applications and…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.