ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?
Abstract
Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.
Read the paper
Related studies
- Climate Change Impacts on Marine Ecosystems
- Assisted Gene Flow to Facilitate Local Adaptation to Climate Change
- Climate Change Effects on Plant Disease: Genomes to Ecosystems
- Biodiversity and Climate Change: Integrating Evolutionary and Ecological Responses of Species and Communities
- A systematic review of climate change education: giving children and young people a ‘voice’ and a ‘hand’ in redressing climate change
- Global Monsoon Dynamics and Climate Change
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.