Evaluating Retrieval-Augmented Generation Strategies with a Virtual Human Application
Abstract
Virtual Humans enhanced with Large Language Models can hold broad conversations, but their answers may sound convincing while still being factually incorrect. Such hallucinations can mislead users and reduce trust, especially because research shows that people often overestimate LLM accuracy and may remain distrustful after errors occur. This loss of trust is particularly concerning in medical contexts, where reliable information is essential. Retrieval-augmented generation (RAG) addresses this issue by grounding LLM responses in external knowledge sources that were not part of the model's training data. The described Virtual Human demonstration system uses RAG with 14 geriatrics patient brochures from the Canisius Wilhelmina Hospital, which were converted into question-answer chunks and embedded. At runtime they were retrieved based on similarity to user queries. The system allows for comparison of four RAG approaches, i.e. Basic RAG, HyDE RAG, Reranking RAG, and Cognitive RAG, each using different strategies to improve retrieval and response quality. Users can choose among these RAG options in the application menu, as well as set some other user preferences. Response quality can be evaluated using RAGAS metrics and latency related data. User experience data can also be collected for correlation to RAG performance data.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.