Evaluating the scientific rigour of climate scenario analysis: LLM-assisted evidence from New Zealand’s mandatory disclosures
Abstract
Abstract Climate change poses systemic risks to financial stability, accelerating global mandates for climate-related financial disclosures. However, while these regulations standardize risk reporting, the scientific rigor of costly corporate scenario exercises remains unexamined. This is acutely evident in New Zealand, where a world-first mandate legally required large entities to disclose their resilience against distinct climate scenarios starting 1 January 2023. This study evaluates the scientific rigor of those disclosures across New Zealand’s first two mandatory reporting periods (FY23/24 and FY24/25). We audited 227 statements using an LLM-assisted Retrieval-Augmented Generation (RAG) framework, validated against expert reviews. Results indicate that while disclosures are increasingly comprehensive, they remain overwhelmingly narrative and limited by highly variable time horizons, model selections, and data sources. This variance limits cross-sector comparability, even within identical industries. Furthermore, while LLMs improve evaluation scalability, they demand precise prompt engineering and continuous ground-truth validation to ensure reliability. Ultimately, these findings expose a critical gap between regulatory compliance and genuine scientific utility. We offer a scalable, automated audit framework for international regulators as global mandates roll out, challenging practitioners to prioritise the methodological consistency required to effectively allocate…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.