Building Conversational Agents for Stroke Rehabilitation: An Evaluation of Large Language Models and Retrieval Augmented Generation
Abstract
Conversational agents (CAs) have been successfully implemented to deliver digital health interventions. However, most of their applications focus on non-communicable diseases, e.g., diabetes. RehabCoach, a CA-based smartphone application designed to assist stroke survivors during at-home unsupervised rehabilitation, utilizes multiple CAs to deliver digital interventions for therapy adherence. Matthias, one of the CAs of RehabCoach, is a novel Large Language Model-based CA that leverages Retrieval Augmented Generation (RAG) to answer questions related to ReHandyBot, an upper limb rehabilitation device. In this work, we assessed the preciseness and conciseness of Matthias’s answers to 14 device-related questions. For each question, Matthias generated 6 answers, based on 3 different prompts and 2 different data sources, i.e., a conversational dataset containing dialogues about how to operate the device and the device manual of instructions. Results from a blinded evaluation performed by two device experts highlight that the responses of the CA generated using the conversational dataset were considered more concise (82,14%) and better overall (60,70%). This work shows the potential of using conversational data for device-assisted rehabilitation CAs deploying RAG.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.