Generation and Automatic Evaluation of Soap Notes from Medical Dialogue Using Large Language Models
Abstract
The creation of nursing records in clinical settings, particularly in the SOAP ($\mathrm{S}, \mathrm{O}, \mathrm{A}$, and P) format, is a significant burden for health-care professionals. Hospitals have huge amounts of clinical data records however, so in this study we propose and validate a method that utilizes Large Language Models (LLMs) to automatically generate and review SOAP-formatted nursing records, leveraging this existing data. The main contributions of this research are twofold. First, we propose a novel approach for improving medical record generation accuracy by combining Retrieval-Augmented Generation (RAG), using a hospital's accumulated historical nursing records as source data, with a reasoning and evidence-citing prompt command to enhance the accuracy of the generation process. Second, we validate the effectiveness of the “LLM-as-a-Judge” framework for evaluating performance during Japanese medical document generation tasks. Our experiments compare the effectiveness of two types of LLMs in various configurations using our proposed methods. The quality of the output is then assessed using hallucination and information omission as evaluation criteria. In this study, we attempt to overcome challenges in the automation of Japanese medical document creation when using LLMs, with the goal of providing guidance for the improvement of the operational efficiency of healthcare professionals and the overall quality of medical care.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.