Assessing the Efficacy of Large Language Models in Health Literacy: A Comprehensive Cross-Sectional Study
Abstract
Enhanced health literacy in children has been empirically linked to better health outcomes over the long term; however, few interventions have been shown to improve health literacy. In this context, we investigate whether large language models (LLMs) can serve as a medium to improve health literacy in children. We tested pediatric conditions using 26 different prompts in ChatGPT-3.5, ChatGPT-4, Microsoft Bing, and Google Bard (now known as Google Gemini). The primary outcome measurement was the reading grade level (RGL) of output as assessed by Gunning Fog, Flesch-Kincaid Grade Level, Automated Readability Index, and Coleman-Liau indices. Word counts were also assessed. Across all models, output for basic prompts such as “Explain” and “What is (are),” were at, or exceeded, the tenth-grade RGL. When prompts were specified to explain conditions from the first- to twelfth-grade level, we found that LLMs had varying abilities to tailor responses based on grade level. ChatGPT-3.5 provided responses that ranged from the seventh-grade to college freshmen RGL while ChatGPT-4 outputted responses from the tenth-grade to the college senior RGL. Microsoft Bing provided responses from the ninth- to eleventh-grade RGL while Google Bard provided responses from the seventh- to tenth-grade RGL. LLMs face challenges in crafting outputs below a sixth-grade RGL. However, their capability to modify outputs above this threshold, provides a potential mechanism for adolescents to explore,…
How strong is the evidence?
Evidence Hub reads this as a cross-sectional, done in people. Its evidence level is 4 of 5 (Low · observational). Observational designs can show an association but not prove cause.
Read the paper
Related studies
- Safety and efficacy of COVID‐19 vaccines: A systematic review and meta‐analysis of controlled and randomized clinical trials
- Accuracy Comparison between Robot-Assisted Dental Implant Placement and Static/Dynamic Computer-Assisted Implant Surgery: A Systematic Review and Meta-Analysis of In Vitro Studies
- Determinants of passive antibody efficacy in SARS-CoV-2 infection: a systematic review and meta-analysis
- Cleaning efficacy of EDDY versus ultrasonically-activated irrigation in root canals: a systematic review and meta-analysis
- Bonding Efficacy of Universal Resin Adhesives to Zirconia Substrates: Systematic Review and Meta-Analysis
- In vitro antimicrobial activity of extracts and essential oils of Cinnamomum, Salvia, and Mentha spp. against foodborne pathogens: A meta-analysis study
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.