Capturing expert uncertainty: ICC-informed soft labelling for volcano-seismicity
Abstract
UNLABELLED: Reliable classification of volcano-seismic signals underpins monitoring and eruption forecasting and is an essential tool for advancing understanding of subsurface processes. However, traditional approaches may overlook the inherent uncertainty and variability between expert judgments. We introduce an innovative method that explicitly quantifies inter-expert agreement using the intraclass correlation coefficient (ICC) and incorporates this measure into probabilistic, ICC-informed soft labels, which can be fed into machine learning pipelines. We conducted a global survey involving 89 experts who classified a set of 80 volcano-seismic events from Ruapehu, New Zealand, providing continuous ratings for standard categories: volcano tectonic (VT), hybrid (HYB), long-period (LP), and other (OT). ICC agreement scores revealed that single-rater scores produce poor agreement between experts even for well-established VT and LP classifications. However, reliability significantly improved for these classifications when multiple expert ratings were combined, although, for HYB and OT categories, expert disagreement remained substantial. We developed a soft labelling methodology that weights class probabilities by their respective ICC scores, resulting in a distribution that naturally reflects expert uncertainty. This demonstrates that ICC-informed soft labels could provide a robust alternative to the hard label standard by explicitly capturing classification uncertainty and…
Read the paper
Related studies
- Current state and future directions for deep learning based automatic seismic fault interpretation: A systematic review
- Physics of Partially Saturated Porous Media: Residual Saturation and Seismic-Wave Propagation
- Modern Imaging Using Seismic Reflection Data
- Effects of marine seismic surveys on free-ranging fauna: a systematic literature review
- A review of the recent seismic activity in the southern part of egypt (upper egypt)
- A review of research on mechanism of seismic attenuation and energy compensation
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.