Outcomes, Evaluation Metrics, and Measurement Tools for Large Language Model Applications Research in Nursing: A Scoping Review
Abstract
Aim: To systematically identify and categorize the outcomes, evaluation metrics, and measurement tools employed in studies evaluating the application of large language models (LLMs) in the nursing field.
Background: LLMs are increasingly applied in nursing education, clinical practice, and research. However, outcomes, evaluation metrics, and measurement tools used to evaluate these applications remain heterogeneous and inconsistently reported, limiting cross-study comparability and evidence synthesis.
Methods: This scoping review was conducted in accordance with the Joanna Briggs Institute methodology and PRISMA-ScR guidelines. Outcomes, evaluation metrics, and measurement tools were extracted and categorized according to the Consensus-based Method for the Selection of Health Outcome Measurement Tools (COMET) framework domains, then mapped to LLM task types based on the Trustworthiness and Reporting of Items for Prediction Model Development-Large Language Models (TRIPOD-LLM) guidelines.
Sources of evidence: Eight English and Chinese databases were searched to identify original empirical studies evaluating LLM applications in nursing, yielding 50 studies that met the inclusion criteria.
Results: A total of 56 distinct evaluative elements (encompassing clinical/educational outcomes, technical evaluation metrics, and standardized measurement tools) were extracted and categorized into three COMET-derived domains: objective efficiency indicators, intermediate effect indicators,…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.