Development and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System
Abstract
OBJECTIVES: This study aimed to develop and evaluate a retrieval-augmented generation (RAG)-based chatbot system designed to optimize hospital operations. By leveraging electronic medical record (EMR) manuals, the system seeks to streamline administrative workflows and enhance healthcare delivery.
METHODS: The system integrated fine-tuned multilingual embedding models (Multilingual-E5-Large and BGE-M3) for indexing and retrieving information from EMR manuals. A dataset comprising 5,931 question-document pairs was constructed through query augmentation and validated by domain experts. Fine-tuning was performed using contrastive learning to enhance semantic understanding, with performance assessed using top-k accuracy metrics. The Solar Mini Chat API was adopted for text generation, prioritizing Korean-language responses and cost efficiency.
RESULTS: The fine-tuned models demonstrated marked improvements in retrieval accuracy, with BGE-M3 achieving 97.6% and Multilingual-E5-Large reaching 89.7%. The chatbot achieved high performance, with query latency under 10 ms and robust retrieval precision, effectively addressing operational EMR queries. Key applications included administrative task support and billing process optimization, highlighting its potential to reduce staff workload and enhance healthcare service delivery.
CONCLUSIONS: The RAG-based chatbot system successfully addressed critical challenges in healthcare administration, improving EMR usability and operational…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.