Evaluating Federated Learning Under Data Heterogeneity: Failure Modes and Practical Mitigations
Abstract
Abstract Federated Learning (FL) allows for collaborative model training across decentralized clients while maintaining data privacy; yet real-world deployments always include heterogeneous and non-identically distributed (non-IID) client data. Existing FL research primarily assesses system performance using aggregate metrics like global or average accuracy, which can hide differences in client behavior. This study provides a thorough empirical evaluation of federated learning under controlled data heterogeneity, with a major emphasis on worst-client performance as a measure of client-level robustness. Experiments are carried out on MNIST and CIFAR-10 utilizing normal FedAvg training and commonly accepted baselines, such as FedProx and q-FedAvg, at various levels of Dirichlet-based label skew. The results show that, as heterogeneity increases, worstclient accuracy falls quickly, revealing silent errors that were previously disguised by aggregate evaluation. A comparison of mitigation options reveals that server-side client re-weighting improves worst-case performance under modest heterogeneity but becomes unstable and damaging under strong non-IID situations. In contrast, simple client-side data balancing reliably improves worst-client accuracy across all heterogeneity regimes without requiring additional infrastructure or algorithmic complexity. These findings emphasize the limitations of aggregate evaluation in federated learning, as well as the need of client-aware…
Read the paper
Related studies
- Active Learning during Federated Learning for Object Detection
- Fairness-aware federated learning
- Decentralized Federated Learning
- Building a Personally Identifiable Information Recognizer in a Privacy Preserved Manner Using Automated Annotation and Federated Learning
- Preservation of Patient Level Privacy: Federated Classification and Calibration Models
- Efficient explainable federated learning
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.