Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study
Abstract
Background There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task. Methods and findings A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset. Models trained on data from either NIH or MSH…
How strong is the evidence?
Evidence Hub reads this as a cross-sectional, done in people. Its evidence level is 4 of 5 (Low · observational). Observational designs can show an association but not prove cause. Reported size: 30,805 participants or records.
Read the paper
Related studies
- Challenges and opportunities with drug repurposing: finding strategies to find alternative uses of therapeutics
- Challenges in Chagas Disease Drug Development
- T Cell Specificity: A Great Challenge in Chagas Disease
- Benznidazole treatment leads to DNA damage in Trypanosoma cruzi and the persistence of rare widely dispersed non-replicative amastigotes in mice
- Rapid Molecular Detection of Terbinafine-resistant Dermatophytes
- Identifying novel drugs with new modes of action for neglected tropical fungal skin diseases (fungal skinNTDs) using an Open Source Drug discovery approach
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.