Large language models for pneumonia detection in radiology reports via text analysis
Abstract
Pneumonia is a common infection in critically ill patients with poor outcomes, and its identification relies heavily on radiological evidence. Most MIMIC-based pneumonia studies rely on structured codes rather than radiological text, which may limit the precision of pneumonia-related case identification. This study evaluated large language models (LLMs) and retrieval-augmented generation (RAG) frameworks for automated detection of radiographic evidence of pneumonia from radiology reports. Radiological reports from the MIMIC-IV database were analyzed using pre-trained models in an inference-only setting without additional training, fine-tuning, or dataset splitting. Model performance was evaluated using accuracy, precision, sensitivity, specificity, and F1-score, with point estimates calculated from the full dataset and corresponding 95% confidence intervals estimated using 1,000 patient-level clustered bootstrap replicates. Independent evaluation was subsequently performed using a GZ dataset from a tertiary hospital in Guangzhou. RAG generally improved model performance, with gemma3-27b achieving the highest accuracy of 96.4% on the MIMIC-IV dataset and maintaining an accuracy of 87.6% on the GZ dataset. Model performance did not increase linearly with parameter size, suggesting that task-specific model suitability may be more important than parameter scale for this application. Although larger models required longer inference times, their increased computational costs did…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.