Toward Trustworthy Chatbots: A Protocol for Red Teaming for Health Related Conversations
Abstract
INTRODUCTION: Health-related chatbots are increasingly used to mediate conversations that carry clinical significance and emotional weight. Retrieval-augmented generation (RAG) can reduce factual errors ("hallucinations"), but the risks remain, with additional challenges coming from chatbots acting against behavioral safety and scope rules. Red teaming, an adversarial testing process that deliberately probes systems for failures before deployment, offers a way to surface potential risks. We describe a task-informed red-teaming protocol for health-related and patient-facing chatbots..
METHODS: Our protocol is composed of an error stratification, single and multi-turn attack evaluation, and a framework for mitigation techniques. We define an error framework that distinguishes Knowledge Adherence (KA: staying faithful to retrieved documents) from Behavioral Adherence (BA: following safety, tone, and scope instructions). Our single-turn attacks consist of seven attack vectors reflecting real-world pressures, including advice-seeking, user distress, and prompt injection. A subset of these vectors are evaluated in multi-turn attacks. We evaluate two mitigation strategies: (1) prompt augmentation, which adds explicit guardrails to the chatbot prompt, and (2) document augmentation, which adds a localized FAQ document to the retrieval corpus. Finally, we apply this protocol to a social care chatbot (specifically supporting Health-Related Social Needs (HRSN)), developed as an agentic…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.