Fine-tuning Bias Neurons for Fair Text-to-Image Generation
Abstract
Diffusion Models (DMs) have revolutionized Text-to-Image (T2I) generation, yet inherent dataset biases often result in skewed representations across demographics, perpetuating stereotypes and social inequities. Existing debiasing approaches primarily focus on the text processing component, overlooking the intricate biases in the diffusion model's U-Net architecture. This paper presents a novel approach to addressing these biases through a causal analysis of bias disentanglement within the U-Net architecture. We introduce the Contrast Neuron Sensitivity Metric, which enables precise identification of neurons sensitive to bias, allowing for targeted interventions. Our debiasing paradigm fine-tunes these identified neurons with a combination of distribution and semantic loss, requiring only 0.2M parameters to be adjusted, which is far less than prior methods. Experiments show that our method effectively removes gender and race biases and maintains the diversity distribution of images. It enables both absolute fairness and relative adjustments by modifying target attribute distributions (e.g., young:old = 7:3). Furthermore, our approach is scalable, allowing simultaneous fine-tuning across multiple biases, and achieves good bias reduction even with non-templated prompts. The code is available on https://github.com/FanQi-AI/Debias.
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.