Construction of a Retrieval-Augmented LLM Agent for Health Education and a Multi-dimensional Evaluation Framework: A Comparative Study of Chinese Large Language Models
Abstract
Large Language Models (LLMs) hold promise for health education but often suffer from factual hallucinations, lack of empathy, and poor readability due to medical jargon.We propose a Health Education Agent System integrating Retrieval-Augmented Generation (RAG) and prompt engineering. We also introduce a five-dimensional evaluation framework (ASREC: Accuracy, Safety, Readability, Empathy, Completeness). The system was tested on a dataset of 500 instructions across five mainstream LLMs (e.g., DeepSeek, Qwen) and evaluated using a hybrid expert and LLM-as-a-Judge approach.The agent architecture significantly outperformed baseline models across all dimensions. Notably, Completeness scores improved by 60.8% and Empathy by 14.3%. Ablation studies confirmed that RAG primarily enhances accuracy and safety, while prompt engineering improves readability and empathy.The proposed agent effectively bridges the gap between professional medical knowledge and public accessibility. The ASREC framework offers a robust tool for evaluating patient-centric medical AI in clinical health education.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.