Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation
Abstract
Diffusion models and flow matching have demonstrated remarkable success in text-to-image generation. While many existing alignment methods primarily focus on fine-tuning pre-trained generative models to maximize a given reward function, these approaches require extensive computational resources and may not generalize well across different objectives. In this work, we propose a novel alignment framework by leveraging the underlying nature of the alignment problem -- sampling from reward-weighted distributions -- and show that it applies to both diffusion models (via score guidance) and flow matching models (via velocity guidance). The score function (velocity field) required for the reward-weighted distribution can be decomposed into the pre-trained score (velocity field) plus a conditional expectation of the reward. For the alignment on the diffusion model, we identify a fundamental challenge: the adversarial nature of the guidance term can introduce undesirable artifacts in the generated images. Therefore, we propose a finetuning-free framework that trains a guidance network to estimate the conditional expectation of the reward. We achieve comparable performance to finetuning-based models with one-step generation with at least a 60% reduction in computational cost. For the alignment on flow matching, we propose a training-free framework that improves the generation quality without additional computational cost.
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.