Enhancing Privacy and Efficiency in Federated Learning: Quantization and Low-Rank Adaptation Approaches
Abstract
Federated Learning (FL) enables collaborative AI model training on decentralized data but faces challenges in ensuring privacy and operational efficiency, especially for large models over wireless networks. This thesis addresses these critical issues by enhancing both aspects in FL systems. Firstly, we provide a rigorous analysis of the interplay between model update quantization and Differential Privacy (DP). Using Rényi Differential Privacy (RDP), we formally demonstrate that lower quantization bit levels can enhance privacy guarantees when applied to Gaussian-noised updates. This theoretical finding, contrary to some prior notions, is empirically validated using Membership Inference Attacks (MIA), confirming that quantization can be a privacy-enhancing mechanism in addition to a communication-saving one. Secondly, we propose a novel framework for differentially private federated fine-tuning (FedFT) of Foundation Models using Low-Rank Adaptation (LoRA). This framework uniquely leverages inherent wireless channel noise as a natural source for DP, reducing the need for artificial noise injection. We analyze the convergence of this split FedFT-LoRA architecture and address the critical problem of noise amplification by proposing a modified LoRA update strategy where only one low-rank matrix is trained while another initialized as scaled orthogonal. Simulations show this approach achieves superior model accuracy under stringent privacy budgets compared to baseline methods.…
Read the paper
Related studies
- Active Learning during Federated Learning for Object Detection
- Fairness-aware federated learning
- Decentralized Federated Learning
- Building a Personally Identifiable Information Recognizer in a Privacy Preserved Manner Using Automated Annotation and Federated Learning
- Preservation of Patient Level Privacy: Federated Classification and Calibration Models
- Efficient explainable federated learning
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.