Safety Mechanisms and Risk Mitigation in Generative AI Mental Health Chatbots: A Systematic Scoping Review
Abstract
Background: Generative AI (GenAI) mental health chatbots are increasingly being developed to help address persistent barriers to mental healthcare. Unlike earlier rule-based and retrieval-based systems, GenAI chatbots generate open-ended outputs that can be inaccurate and unsafe. Documented harms from general-purpose GenAI chatbots have highlighted the need for purpose-built interventions with dedicated safeguards, yet how safety is implemented in such interventions remains poorly understood.
Methods: This scoping review followed the Joanna Briggs Institute methodology and PRISMA-ScR guidelines, with a prospectively registered and peer-reviewed protocol. A systematic search of seven academic databases and search engines including MEDLINE, Scopus, PsycINFO, ACM Digital Library, IEEE Xplore, Google Scholar and Consensus was conducted in July 2025. Two reviewers independently screened records and extracted data. Safety mechanisms and risk mitigation strategies were narratively synthesised across three pre-specified domains: technical safeguards, pre-deployment safety considerations, and delivery-phase risk mitigation strategies.
Results: Twenty-one studies across 11 countries were included. Most interventions incorporated at least one technical safety mechanism, most commonly fine-tuning and prompt engineering. A smaller subset implemented layered safety architectures combining retrieval systems, content filters or risk classifiers, and rule-based algorithms. Pre-deployment…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.